Jul 09 23:28:01.718990 ip-10-0-140-8 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 23:28:02.174822 ip-10-0-140-8 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 23:28:02.174822 ip-10-0-140-8 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 23:28:02.174822 ip-10-0-140-8 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 23:28:02.174822 ip-10-0-140-8 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 23:28:02.174822 ip-10-0-140-8 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 23:28:02.176761 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.176666    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 23:28:02.182671 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182638    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 23:28:02.182671 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182669    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 23:28:02.182671 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182673    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182676    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182681    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182684    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182687    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182689    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182693    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182695    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182698    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182702    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182705    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182707    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182710    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182713    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182716    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182719    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182721    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182726    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182731    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182734    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 23:28:02.182789 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182737    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182740    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182742    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182745    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182748    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182750    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182753    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182756    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182758    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182761    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182764    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182766    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182769    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182771    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182779    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182782    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182785    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182788    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182791    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182794    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 23:28:02.183258 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182796    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182800    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182805    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182807    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182811    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182814    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182817    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182820    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182822    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182825    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182828    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182831    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182833    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182836    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182838    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182841    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182843    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182846    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182848    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 23:28:02.183906 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182851    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182853    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182856    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182859    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182861    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182864    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182866    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182870    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182873    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182876    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182879    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182882    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182884    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182887    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182890    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182892    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182895    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182897    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182900    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 23:28:02.184587 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182904    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182907    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182912    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182915    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182919    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182921    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182924    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182927    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182929    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182932    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182934    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182937    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182940    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182943    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182945    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182948    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182950    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182953    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.182955    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183721    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 23:28:02.185309 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183731    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183735    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183737    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183743    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183746    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183750    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183752    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183755    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183758    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183761    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183764    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183766    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183769    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183772    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183774    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183778    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183782    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183785    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183788    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 23:28:02.186038 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183791    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183794    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183797    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183799    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183802    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183805    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183807    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183810    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183812    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183815    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183818    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183822    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183824    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183827    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183830    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183833    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183835    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183838    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183840    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183843    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 23:28:02.186785 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183855    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183858    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183861    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183864    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183867    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183869    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183872    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183874    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183877    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183880    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183883    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183887    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183889    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183892    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183894    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183897    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183900    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183902    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183905    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183907    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183910    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 23:28:02.187514 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183912    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183915    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183918    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183921    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183923    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183927    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183929    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183932    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183935    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183938    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183941    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183943    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183945    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183948    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183951    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183953    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183956    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183958    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183961    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183963    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 23:28:02.188292 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183966    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183969    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183971    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183974    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183976    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183979    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183981    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183984    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183987    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183990    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183992    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183995    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.183997    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184000    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184002    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184005    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184007    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184010    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184402    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 23:28:02.189012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184406    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184409    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184412    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184415    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184418    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184420    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184423    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184426    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184428    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184431    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184434    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184436    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184439    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184442    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184445    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184447    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184450    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184452    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184455    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184458    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 23:28:02.189722 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184462    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184465    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184468    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184471    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184474    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184476    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184479    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184482    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184484    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184490    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184492    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184495    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184498    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184501    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184505    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184508    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184511    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184513    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184516    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184518    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 23:28:02.190445 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184521    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184523    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184526    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184528    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184530    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184534    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184537    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184539    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184542    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184544    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184547    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184549    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184552    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184554    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184557    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184559    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184562    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184565    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184567    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184570    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 23:28:02.191177 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184572    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184576    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184579    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184581    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184584    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184587    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184589    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184592    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184595    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184598    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184600    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184604    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184607    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184609    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184612    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184614    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184616    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184619    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184622    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 23:28:02.192034 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184624    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184627    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184630    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184632    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184634    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184637    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184640    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184642    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184645    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184647    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184664    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184666    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184669    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184673    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184677    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184681    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184684    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184687    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.184690    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184772    2580 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184784    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 23:28:02.192724 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184791    2580 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184796    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184800    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184803    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184808    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184812    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184815    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184818    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184823    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184826    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184829    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184833    2580 flags.go:64] FLAG: --cgroup-root=""
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184836    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184839    2580 flags.go:64] FLAG: --client-ca-file=""
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184842    2580 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184845    2580 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184849    2580 flags.go:64] FLAG: --cluster-domain=""
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184852    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184855    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184858    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184862    2580 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184867    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184870    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 23:28:02.193467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184874    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184877    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184881    2580 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184884    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184887    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184890    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184893    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184898    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184901    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184904    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184906    2580 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184910    2580 flags.go:64] FLAG: --enable-server="true"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184913    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184918    2580 flags.go:64] FLAG: --event-burst="100"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184921    2580 flags.go:64] FLAG: --event-qps="50"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184924    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184927    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184930    2580 flags.go:64] FLAG: --eviction-hard=""
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184938    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184941    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184944    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184947    2580 flags.go:64] FLAG: --eviction-soft=""
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184950    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184953    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184956    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 23:28:02.194342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184959    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184963    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184966    2580 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184969    2580 flags.go:64] FLAG: --feature-gates=""
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184972    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184975    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184978    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184982    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184985    2580 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184988    2580 flags.go:64] FLAG: --help="false"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184991    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184994    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.184997    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185000    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185003    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185006    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185010    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185012    2580 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185015    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185372    2580 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185376    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185379    2580 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185382    2580 flags.go:64] FLAG: --kube-reserved=""
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185385    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 23:28:02.195269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185388    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185392    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185395    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185398    2580 flags.go:64] FLAG: --lock-file=""
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185401    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185404    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185407    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185412    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185415    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185418    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185421    2580 flags.go:64] FLAG: --logging-format="text"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185424    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185428    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185430    2580 flags.go:64] FLAG: --manifest-url=""
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185433    2580 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185437    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185440    2580 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185445    2580 flags.go:64] FLAG: --max-pods="110"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185448    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185451    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185455    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185458    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185461    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185464    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185467    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 23:28:02.196155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185474    2580 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185477    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185480    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185484    2580 flags.go:64] FLAG: --pod-cidr=""
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185487    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185493    2580 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185496    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185499    2580 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185503    2580 flags.go:64] FLAG: --port="10250"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185506    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185509    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09071974b7c03737f"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185512    2580 flags.go:64] FLAG: --qos-reserved=""
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185515    2580 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185518    2580 flags.go:64] FLAG: --register-node="true"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185520    2580 flags.go:64] FLAG: --register-with-taints=""
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185524    2580 flags.go:64] FLAG: --registry-burst="10"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185527    2580 flags.go:64] FLAG: --registry-qps="5"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185530    2580 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185532    2580 flags.go:64] FLAG: --reserved-memory=""
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185536    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185539    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185542    2580 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185545    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185548    2580 flags.go:64] FLAG: --runonce="false"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185551    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 23:28:02.197083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185556    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185559    2580 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185561    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185565    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185568    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185570    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185574    2580 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185576    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185579    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185582    2580 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185585    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185588    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185591    2580 flags.go:64] FLAG: --system-cgroups=""
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185594    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185599    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185603    2580 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185606    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185609    2580 flags.go:64] FLAG: --tls-min-version=""
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185612    2580 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185614    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185617    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185620    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185623    2580 flags.go:64] FLAG: --v="2"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185627    2580 flags.go:64] FLAG: --version="false"
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185632    2580 flags.go:64] FLAG: --vmodule=""
Jul 09 23:28:02.197992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185636    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.185639    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185767    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185771    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185774    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185777    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185780    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185784    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185787    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185789    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185792    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185794    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185797    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185800    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185802    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185805    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185807    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185810    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185813    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185816    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 23:28:02.198954 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185818    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185820    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185823    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185826    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185829    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185831    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185834    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185836    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185838    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185841    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185843    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185846    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185849    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185851    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185854    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185856    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185858    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185861    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185863    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185868    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 23:28:02.199717 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185871    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185873    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185876    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185879    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185881    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185884    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185887    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185889    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185891    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185896    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185899    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185902    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185905    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185909    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185913    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185915    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185918    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185920    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185922    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 23:28:02.200444 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185925    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185929    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185933    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185936    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185938    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185941    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185943    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185946    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185948    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185951    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185953    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185956    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185960    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185962    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185965    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185968    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185970    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185973    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185975    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 23:28:02.201140 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185978    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185981    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185983    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185986    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185988    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185991    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185994    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.185997    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186001    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186005    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186008    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186011    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186014    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186017    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186020    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186022    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186025    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186028    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186031    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186034    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 23:28:02.201854 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186036    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186039    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.186042    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.186800    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.195450    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.195489    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.196273    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.196282    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196347    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196354    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196358    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196361    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196364    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196367    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196370    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 23:28:02.202601 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196373    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196376    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196378    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196381    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196384    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196386    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196389    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196391    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196394    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196397    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196399    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196402    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196405    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196408    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196410    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196413    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196415    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196418    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196421    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196423    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 23:28:02.203173 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196427    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196430    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196433    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196436    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196440    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196443    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196445    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196448    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196450    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196453    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196456    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196458    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196461    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196463    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196466    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196469    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196471    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196474    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196477    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196480    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 23:28:02.203933 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196483    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196486    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196488    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196491    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196493    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196496    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196498    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196501    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196503    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196506    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196508    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196511    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196516    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196519    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196523    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196526    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196529    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196532    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196534    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196537    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 23:28:02.204644 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196540    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196542    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196545    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196549    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196553    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196555    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196559    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196561    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196565    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196567    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196570    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196574    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196578    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196582    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196585    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196588    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196591    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196593    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196596    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 23:28:02.205382 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196598    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196601    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196604    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196606    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196609    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196612    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196614    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196618    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196620    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196623    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196626    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196629    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196632    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.196637    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196774    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 23:28:02.206066 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196780    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196783    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196786    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196790    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196792    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196795    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196798    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196801    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196803    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196806    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196809    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196813    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196816    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196818    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196821    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196823    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196826    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196829    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196831    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196834    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 23:28:02.206607 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196836    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196839    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196841    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196844    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196848    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196852    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196854    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196857    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196861    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196866    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196869    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196872    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196874    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196877    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196879    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196882    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196885    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196887    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196890    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 23:28:02.207474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196892    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196895    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196897    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196900    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196904    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196908    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196910    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196913    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196916    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196918    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196921    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196923    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196926    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196928    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196931    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196933    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196936    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196939    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196943    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 23:28:02.208162 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196947    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196950    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196953    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196956    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196959    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196962    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196964    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196967    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196969    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196972    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196974    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196977    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196980    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196982    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196985    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196987    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196990    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196992    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196995    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.196998    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 23:28:02.208857 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197000    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197003    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197006    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197008    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197011    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197013    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197016    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197019    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197021    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197024    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197026    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197029    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197032    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197034    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197037    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197040    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197042    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197045    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197047    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 23:28:02.209548 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:02.197050    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 23:28:02.210236 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.197055    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 23:28:02.210236 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.197192    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 23:28:02.210236 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.199705    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 23:28:02.210236 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.201177    2580 server.go:1019] "Starting client certificate rotation"
Jul 09 23:28:02.210236 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.201290    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 23:28:02.210236 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.201322    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 23:28:02.210236 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.204204    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 23:28:02.210236 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.205409    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 23:28:02.221912 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.221885    2580 log.go:25] "Validated CRI v1 runtime API"
Jul 09 23:28:02.225362 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.225339    2580 log.go:25] "Validated CRI v1 image API"
Jul 09 23:28:02.226255 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.226235    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 23:28:02.229793 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.229767    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d16630fd-cc26-48ed-b992-76acb49469b5:/dev/nvme0n1p3 d27b3524-6716-412e-a747-7624a8f6c49a:/dev/nvme0n1p4]
Jul 09 23:28:02.229873 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.229790    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 23:28:02.231314 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.231291    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 23:28:02.235504 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.235383    2580 manager.go:216] Machine: {Timestamp:2026-07-09 23:28:02.234884239 +0000 UTC m=+0.378815017 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100053 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29ae45a5fdad54a81712b73acbb1e5 SystemUUID:ec29ae45-a5fd-ad54-a817-12b73acbb1e5 BootID:d7599647-03ac-46d5-b011-796ded0adde9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f9:f6:42:c4:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f9:f6:42:c4:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:89:ef:e0:d4:58 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 23:28:02.235504 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.235495    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 23:28:02.235675 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.235583    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 23:28:02.241168 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.240969    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 23:28:02.241324 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.241174    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-8.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 23:28:02.241994 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.241980    2580 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 23:28:02.241994 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.241994    2580 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 23:28:02.242075 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.242002    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 23:28:02.242946 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.242933    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 23:28:02.244064 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.244052    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 23:28:02.247736 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.247720    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 23:28:02.247892 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.247880    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 23:28:02.247948 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.247916    2580 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 23:28:02.247948 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.247930    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 23:28:02.248027 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.247950    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 23:28:02.248027 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.247963    2580 kubelet.go:398] "Adding apiserver pod source"
Jul 09 23:28:02.248027 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.247976    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 23:28:02.248462 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.248449    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 23:28:02.248515 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.248474    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 23:28:02.249545 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.249519    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fnfj4"
Jul 09 23:28:02.251084 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.251067    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 23:28:02.252689 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.252673    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 23:28:02.252774 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.252704    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 23:28:02.253447 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.253436    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 23:28:02.253494 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.253454    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 23:28:02.253494 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.253460    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 23:28:02.253494 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.253466    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 23:28:02.253494 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.253474    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 23:28:02.253494 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.253481    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 23:28:02.253663 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.253518    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 23:28:02.253663 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.253524    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 23:28:02.253663 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.253531    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 23:28:02.253663 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.253536    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 23:28:02.253663 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.253545    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 23:28:02.253663 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.253553    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 23:28:02.254353 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.254342    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 23:28:02.254353 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.254354    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 23:28:02.256187 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.256153    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 23:28:02.256187 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.256166    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 23:28:02.256387 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.256355    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 23:28:02.257736 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.257723    2580 server.go:1268] "Started kubelet"
Jul 09 23:28:02.257892 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.257813    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 23:28:02.257989 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.257923    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 23:28:02.257989 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.257854    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 23:28:02.258684 ip-10-0-140-8 systemd[1]: Started Kubernetes Kubelet.
Jul 09 23:28:02.259584 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.259544    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 23:28:02.260645 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.260628    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fnfj4"
Jul 09 23:28:02.260839 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.260826    2580 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 23:28:02.264518 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.264496    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 23:28:02.266813 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.266794    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 23:28:02.266897 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.266796    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 23:28:02.267015 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.266997    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:02.267615 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.267565    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 23:28:02.267615 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.267593    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 23:28:02.267615 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.267601    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 23:28:02.267830 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.267789    2580 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 23:28:02.267830 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.267797    2580 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 23:28:02.267935 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.267908    2580 factory.go:55] Registering systemd factory
Jul 09 23:28:02.267935 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.267935    2580 factory.go:223] Registration of the systemd container factory successfully
Jul 09 23:28:02.268290 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.268270    2580 factory.go:153] Registering CRI-O factory
Jul 09 23:28:02.268382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.268293    2580 factory.go:223] Registration of the crio container factory successfully
Jul 09 23:28:02.268437 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.268369    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 23:28:02.268437 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.268412    2580 factory.go:103] Registering Raw factory
Jul 09 23:28:02.268530 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.268470    2580 manager.go:1196] Started watching for new ooms in manager
Jul 09 23:28:02.269457 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.269437    2580 manager.go:349] Starting recovery of all containers
Jul 09 23:28:02.271700 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.271679    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 23:28:02.275511 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.275482    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-8.ec2.internal\" not found" node="ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.285492 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.285465    2580 manager.go:354] Recovery completed
Jul 09 23:28:02.289967 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.289948    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 23:28:02.292184 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.292164    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 23:28:02.292292 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.292194    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 23:28:02.292292 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.292205    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID"
Jul 09 23:28:02.292757 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.292740    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 23:28:02.292792 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.292757    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 23:28:02.292792 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.292774    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 23:28:02.292860 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.292810    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 23:28:02.293893 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.293875    2580 policy_none.go:49] "None policy: Start"
Jul 09 23:28:02.293893 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.293894    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 23:28:02.294016 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.293905    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 23:28:02.294539 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.294522    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 23:28:02.294615 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.294556    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 23:28:02.294615 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.294572    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID"
Jul 09 23:28:02.294885 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.294872    2580 policy_none.go:47] "Start"
Jul 09 23:28:02.326668 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.326451    2580 manager.go:337] "Starting Device Plugin manager"
Jul 09 23:28:02.326818 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.326695    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 23:28:02.326818 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.326709    2580 server.go:87] "Starting device plugin registration server"
Jul 09 23:28:02.326995 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.326982    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 23:28:02.327043 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.327000    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 23:28:02.327125 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.327102    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 23:28:02.327286 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.327194    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 23:28:02.327286 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.327206    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 23:28:02.327899 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.327877    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 23:28:02.327981 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.327943    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:02.411177 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.411136    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 23:28:02.412428 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.412416    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 23:28:02.412516 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.412451    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 23:28:02.412516 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.412491    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 23:28:02.412619 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.412527    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 23:28:02.414886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.414858    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 23:28:02.427714 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.427686    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 23:28:02.428643 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.428624    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 23:28:02.428772 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.428673    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 23:28:02.428772 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.428688    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID"
Jul 09 23:28:02.428772 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.428718    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.439302 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.439276    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.439302 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.439307    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-8.ec2.internal\": node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:02.454349 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.454316    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:02.513630 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.513582    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal"]
Jul 09 23:28:02.513844 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.513696    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 23:28:02.515694 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.515679    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 23:28:02.515771 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.515710    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 23:28:02.515771 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.515724    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID"
Jul 09 23:28:02.516921 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.516908    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 23:28:02.517047 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.517033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.517102 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.517093    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 23:28:02.518088 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.518071    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 23:28:02.518169 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.518102    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 23:28:02.518169 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.518104    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 23:28:02.518169 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.518144    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 23:28:02.518169 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.518154    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID"
Jul 09 23:28:02.518288 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.518117    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID"
Jul 09 23:28:02.519146 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.519129    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.519209 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.519154    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 23:28:02.519865 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.519839    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 23:28:02.519957 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.519883    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 23:28:02.519957 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.519898    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeHasSufficientPID"
Jul 09 23:28:02.548575 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.548547    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-8.ec2.internal\" not found" node="ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.552957 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.552941    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-8.ec2.internal\" not found" node="ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.554857 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.554844    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:02.568519 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.568491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2584345ba2d507ebd069c4f5d2f9d4a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal\" (UID: \"2584345ba2d507ebd069c4f5d2f9d4a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.568601 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.568522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2584345ba2d507ebd069c4f5d2f9d4a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal\" (UID: \"2584345ba2d507ebd069c4f5d2f9d4a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.568601 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.568543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3203ef60a382e0f7b0ec847ea9d1892c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-8.ec2.internal\" (UID: \"3203ef60a382e0f7b0ec847ea9d1892c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.655800 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.655770    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:02.669114 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.669056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2584345ba2d507ebd069c4f5d2f9d4a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal\" (UID: \"2584345ba2d507ebd069c4f5d2f9d4a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.669114 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.669092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3203ef60a382e0f7b0ec847ea9d1892c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-8.ec2.internal\" (UID: \"3203ef60a382e0f7b0ec847ea9d1892c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.669243 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.669125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2584345ba2d507ebd069c4f5d2f9d4a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal\" (UID: \"2584345ba2d507ebd069c4f5d2f9d4a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.669243 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.669152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2584345ba2d507ebd069c4f5d2f9d4a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal\" (UID: \"2584345ba2d507ebd069c4f5d2f9d4a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.669243 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.669159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3203ef60a382e0f7b0ec847ea9d1892c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-8.ec2.internal\" (UID: \"3203ef60a382e0f7b0ec847ea9d1892c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.669243 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.669156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2584345ba2d507ebd069c4f5d2f9d4a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal\" (UID: \"2584345ba2d507ebd069c4f5d2f9d4a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.756973 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.756940    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:02.852853 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.852808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.855068 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:02.855045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:02.857361 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.857149    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:02.957349 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:02.957258    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:03.057899 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:03.057861    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:03.164375 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:03.164335    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:03.200543 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.200511    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 23:28:03.201120 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.200662    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 23:28:03.201120 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.200724    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 23:28:03.265104 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:03.265026    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:03.267145 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.267109    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 23:23:02 +0000 UTC" deadline="2027-12-27 18:57:36.387303934 +0000 UTC"
Jul 09 23:28:03.267231 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.267145    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12859h29m33.120163559s"
Jul 09 23:28:03.356276 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:03.356239    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2584345ba2d507ebd069c4f5d2f9d4a9.slice/crio-88f43445f6629baa2375973bc28b2c7fdd73f441939dcae4af5dbc2e45726871 WatchSource:0}: Error finding container 88f43445f6629baa2375973bc28b2c7fdd73f441939dcae4af5dbc2e45726871: Status 404 returned error can't find the container with id 88f43445f6629baa2375973bc28b2c7fdd73f441939dcae4af5dbc2e45726871
Jul 09 23:28:03.362477 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.362458    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 23:28:03.364600 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:03.364571    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3203ef60a382e0f7b0ec847ea9d1892c.slice/crio-b552d83579602d18d47e51829f6d139004ceb8dde379b02857c870f7d3872ffb WatchSource:0}: Error finding container b552d83579602d18d47e51829f6d139004ceb8dde379b02857c870f7d3872ffb: Status 404 returned error can't find the container with id b552d83579602d18d47e51829f6d139004ceb8dde379b02857c870f7d3872ffb
Jul 09 23:28:03.365263 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:03.365240    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:03.416893 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.416338    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" event={"ID":"2584345ba2d507ebd069c4f5d2f9d4a9","Type":"ContainerStarted","Data":"88f43445f6629baa2375973bc28b2c7fdd73f441939dcae4af5dbc2e45726871"}
Jul 09 23:28:03.417565 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.417525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal" event={"ID":"3203ef60a382e0f7b0ec847ea9d1892c","Type":"ContainerStarted","Data":"b552d83579602d18d47e51829f6d139004ceb8dde379b02857c870f7d3872ffb"}
Jul 09 23:28:03.465892 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:03.465852    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:03.550635 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.550537    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 23:28:03.566664 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:03.566622    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:03.666827 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:03.666785    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-8.ec2.internal\" not found"
Jul 09 23:28:03.758074 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.758040    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 23:28:03.767417 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.767390    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:03.778414 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.778384    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 23:28:03.779607 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.779574    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal"
Jul 09 23:28:03.788046 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:03.787992    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 23:28:04.168877 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.168840    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 23:28:04.248847 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.248810    2580 apiserver.go:52] "Watching apiserver"
Jul 09 23:28:04.256907 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.256875    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 23:28:04.257368 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.257337    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-jrmhz","openshift-multus/multus-additional-cni-plugins-p96p7","openshift-multus/multus-grfjv","openshift-multus/network-metrics-daemon-m5bj7","openshift-network-diagnostics/network-check-target-hgsp8","kube-system/konnectivity-agent-c7j2c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q","openshift-dns/node-resolver-2lhxc","openshift-image-registry/node-ca-4tjx2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal","openshift-network-operator/iptables-alerter-6vbbs","openshift-ovn-kubernetes/ovnkube-node-rvp85","kube-system/global-pull-secret-syncer-tmmgg","kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal"]
Jul 09 23:28:04.259041 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.259019    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:04.259156 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.259107    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:04.260234 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.260210    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.261518 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.261441    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.263180 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.262900    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 23:28:04.263180 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.262993    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 23:28:04.263180 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.262997    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 23:28:04.263180 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.262999    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 23:28:04.263180 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.263138    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6pxbm\""
Jul 09 23:28:04.263490 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.263214    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 23:28:04.263490 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.263327    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.263598 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.263587    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p6nbh\""
Jul 09 23:28:04.263864 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.263846    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 23:28:04.263864 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.263846    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 23:28:04.264045 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.264018    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 23:28:04.264750 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.264729    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:04.264840 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.264806    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:04.266773 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.266753    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h95m7\""
Jul 09 23:28:04.267315 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.266985    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c7j2c"
Jul 09 23:28:04.267315 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.267142    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 23:28:04.267315 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.267198    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 23:28:04.268092 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.268068    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.270522 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.270038    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2lhxc"
Jul 09 23:28:04.270522 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.270114    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 23:28:04.270522 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.270189    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gftfg\""
Jul 09 23:28:04.270522 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.270221    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 23:28:04.270522 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.270416    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 23:28:04.271089 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.271069    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4tjx2"
Jul 09 23:28:04.271700 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.271647    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jd2pn\""
Jul 09 23:28:04.271802 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.271719    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 23:28:04.272542 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.272519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6vbbs"
Jul 09 23:28:04.272917 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.272897    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 23:28:04.272990 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.272929    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bc2nc\""
Jul 09 23:28:04.273181 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.273163    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 23:28:04.273640 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.273544    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 23:28:04.274245 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.274117    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 23:28:04.274245 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.274177    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4mm5x\""
Jul 09 23:28:04.274421 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.274393    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 23:28:04.274421 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.274418    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.275174 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.275062    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 23:28:04.277765 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.276395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:04.277765 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.276479    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:04.277765 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.277503    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fx5jx\""
Jul 09 23:28:04.277765 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.277748    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 23:28:04.278019 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.277936    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 23:28:04.278238 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.278103    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 23:28:04.278822 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.278579    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4fc9c\""
Jul 09 23:28:04.278822 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.278632    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.279530    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.279557    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.279898    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-multus-conf-dir\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/76444aa9-03ec-4980-97f4-dec9f17bbdc6-multus-daemon-config\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-modprobe-d\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-sysctl-conf\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-sys\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-var-lib-kubelet\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55kl2\" (UniqueName: \"kubernetes.io/projected/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-kube-api-access-55kl2\") pod \"network-metrics-daemon-m5bj7\" (UID: \"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8\") " pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/874f202e-075e-49fd-b6c9-2729f95eb09e-agent-certs\") pod \"konnectivity-agent-c7j2c\" (UID: \"874f202e-075e-49fd-b6c9-2729f95eb09e\") " pod="kube-system/konnectivity-agent-c7j2c"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-cni-binary-copy\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-system-cni-dir\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-run-netns\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-etc-kubernetes\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98nmb\" (UniqueName: \"kubernetes.io/projected/76444aa9-03ec-4980-97f4-dec9f17bbdc6-kube-api-access-98nmb\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.281233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-run\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.282153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqffh\" (UniqueName: \"kubernetes.io/projected/6592a0ab-d648-4559-aebe-a8bbf9462e1b-kube-api-access-rqffh\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.282153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280820    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-etc-selinux\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.282153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-multus-cni-dir\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.282153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76444aa9-03ec-4980-97f4-dec9f17bbdc6-cni-binary-copy\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.282153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/874f202e-075e-49fd-b6c9-2729f95eb09e-konnectivity-ca\") pod \"konnectivity-agent-c7j2c\" (UID: \"874f202e-075e-49fd-b6c9-2729f95eb09e\") " pod="kube-system/konnectivity-agent-c7j2c"
Jul 09 23:28:04.282153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-device-dir\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.282153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.280988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-registration-dir\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.282153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.281469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-os-release\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.282153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.281474    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 23:28:04.282153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.281550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.282153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.281960    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 23:28:04.282710 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-run-multus-certs\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.282710 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-sysctl-d\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.282710 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-lib-modules\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.282710 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9869t\" (UniqueName: \"kubernetes.io/projected/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-kube-api-access-9869t\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.282710 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-multus-socket-dir-parent\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.282710 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-var-lib-kubelet\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.282710 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-systemd\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.282710 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-tuned\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.283141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6592a0ab-d648-4559-aebe-a8bbf9462e1b-tmp\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.283141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs\") pod \"network-metrics-daemon-m5bj7\" (UID: \"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8\") " pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:04.283141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-cnibin\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.283141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-run-k8s-cni-cncf-io\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.283141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-var-lib-cni-bin\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.283141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.282979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-sysconfig\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.283141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.283014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-host\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.283141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.283059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-system-cni-dir\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.283141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.283102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.283831 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.283810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-sys-fs\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.283901 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.283850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4jcl\" (UniqueName: \"kubernetes.io/projected/20917dba-8a4d-4cd2-9974-5147e5894bd5-kube-api-access-f4jcl\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.283901 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.283881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdwx\" (UniqueName: \"kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx\") pod \"network-check-target-hgsp8\" (UID: \"d10a13cc-9e45-49e7-a53b-e03df17e48e7\") " pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:04.283965 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.283909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-kubernetes\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.283965 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.283937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-os-release\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.284031 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.283984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-var-lib-cni-multus\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.284031 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.284014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-socket-dir\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.284093 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.284044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-cnibin\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.284093 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.284071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.284177 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.284099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-hostroot\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.303860 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.303833    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wt2pq"
Jul 09 23:28:04.311704 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.311679    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wt2pq"
Jul 09 23:28:04.368674 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.368628    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 23:28:04.384694 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.384646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-multus-socket-dir-parent\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.384861 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.384699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-systemd\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.384861 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.384728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6592a0ab-d648-4559-aebe-a8bbf9462e1b-tmp\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.384861 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.384752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-run-netns\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.384861 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.384753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-multus-socket-dir-parent\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.384861 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.384777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20b8cf45-3b5c-46f1-adb0-ebebf097e6e6-host\") pod \"node-ca-4tjx2\" (UID: \"20b8cf45-3b5c-46f1-adb0-ebebf097e6e6\") " pod="openshift-image-registry/node-ca-4tjx2"
Jul 09 23:28:04.384861 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.384802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-sysconfig\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.384861 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.384825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-host\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.384861 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.384838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-systemd\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.384861 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.384849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww8vh\" (UniqueName: \"kubernetes.io/projected/febddf31-55fc-4ec0-9ebb-019c5952772d-kube-api-access-ww8vh\") pod \"iptables-alerter-6vbbs\" (UID: \"febddf31-55fc-4ec0-9ebb-019c5952772d\") " pod="openshift-network-operator/iptables-alerter-6vbbs"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.384890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-sysconfig\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.384907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-host\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.384961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-run-openvswitch\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4jcl\" (UniqueName: \"kubernetes.io/projected/20917dba-8a4d-4cd2-9974-5147e5894bd5-kube-api-access-f4jcl\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdwx\" (UniqueName: \"kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx\") pod \"network-check-target-hgsp8\" (UID: \"d10a13cc-9e45-49e7-a53b-e03df17e48e7\") " pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdnqv\" (UniqueName: \"kubernetes.io/projected/2dabdc4e-dedb-449e-91eb-56201e293152-kube-api-access-pdnqv\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-sys\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385122    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-var-lib-kubelet\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-var-lib-cni-multus\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-cnibin\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-multus-conf-dir\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-slash\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-cni-netd\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-modprobe-d\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-sysctl-conf\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.385334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-var-lib-cni-multus\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kl2\" (UniqueName: \"kubernetes.io/projected/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-kube-api-access-55kl2\") pod \"network-metrics-daemon-m5bj7\" (UID: \"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8\") " pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385392    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-var-lib-kubelet\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-sys\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/874f202e-075e-49fd-b6c9-2729f95eb09e-agent-certs\") pod \"konnectivity-agent-c7j2c\" (UID: \"874f202e-075e-49fd-b6c9-2729f95eb09e\") " pod="kube-system/konnectivity-agent-c7j2c"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/febddf31-55fc-4ec0-9ebb-019c5952772d-host-slash\") pod \"iptables-alerter-6vbbs\" (UID: \"febddf31-55fc-4ec0-9ebb-019c5952772d\") " pod="openshift-network-operator/iptables-alerter-6vbbs"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-multus-conf-dir\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-var-lib-openvswitch\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-cnibin\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb-tmp-dir\") pod \"node-resolver-2lhxc\" (UID: \"53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb\") " pod="openshift-dns/node-resolver-2lhxc"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-cni-binary-copy\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-system-cni-dir\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385808    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-sysctl-conf\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-run-netns\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-run\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-kubelet\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.386172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-run-netns\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-modprobe-d\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fe6a62a4-df2a-4d08-b134-213d033b0e28-kubelet-config\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-node-log\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.385996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-system-cni-dir\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386036    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-run\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dabdc4e-dedb-449e-91eb-56201e293152-env-overrides\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-device-dir\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-os-release\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-device-dir\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-run-multus-certs\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs\") pod \"network-metrics-daemon-m5bj7\" (UID: \"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8\") " pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-os-release\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.387055 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-run-multus-certs\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.386450    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386447    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9869t\" (UniqueName: \"kubernetes.io/projected/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-kube-api-access-9869t\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-var-lib-kubelet\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386529    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-tuned\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-cnibin\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-run-k8s-cni-cncf-io\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-var-lib-kubelet\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-var-lib-cni-bin\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-cni-binary-copy\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-var-lib-cni-bin\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/febddf31-55fc-4ec0-9ebb-019c5952772d-iptables-alerter-script\") pod \"iptables-alerter-6vbbs\" (UID: \"febddf31-55fc-4ec0-9ebb-019c5952772d\") " pod="openshift-network-operator/iptables-alerter-6vbbs"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-cnibin\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-system-cni-dir\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-host-run-k8s-cni-cncf-io\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-system-cni-dir\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-sys-fs\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.387886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.386836    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-sys-fs\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.388672 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.387260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.388672 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.387344    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs podName:1e7add64-a9cc-4acc-9102-a8c5efbbc5a8 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:04.886806766 +0000 UTC m=+3.030737555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs") pod "network-metrics-daemon-m5bj7" (UID: "1e7add64-a9cc-4acc-9102-a8c5efbbc5a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:04.388672 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.387382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-kubernetes\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.388672 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.387468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-kubernetes\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.388672 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.387512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-systemd-units\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.388672 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.387540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fe6a62a4-df2a-4d08-b134-213d033b0e28-dbus\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:04.388672 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.387606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/76444aa9-03ec-4980-97f4-dec9f17bbdc6-multus-daemon-config\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.388672 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/76444aa9-03ec-4980-97f4-dec9f17bbdc6-multus-daemon-config\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.388672 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.387634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-os-release\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.388672 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-os-release\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.388672 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-socket-dir\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.388672 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-hostroot\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-hostroot\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dabdc4e-dedb-449e-91eb-56201e293152-ovn-node-metrics-cert\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-run-ovn-kubernetes\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrkg\" (UniqueName: \"kubernetes.io/projected/53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb-kube-api-access-fkrkg\") pod \"node-resolver-2lhxc\" (UID: \"53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb\") " pod="openshift-dns/node-resolver-2lhxc"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-etc-kubernetes\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98nmb\" (UniqueName: \"kubernetes.io/projected/76444aa9-03ec-4980-97f4-dec9f17bbdc6-kube-api-access-98nmb\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqffh\" (UniqueName: \"kubernetes.io/projected/6592a0ab-d648-4559-aebe-a8bbf9462e1b-kube-api-access-rqffh\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/874f202e-075e-49fd-b6c9-2729f95eb09e-konnectivity-ca\") pod \"konnectivity-agent-c7j2c\" (UID: \"874f202e-075e-49fd-b6c9-2729f95eb09e\") " pod="kube-system/konnectivity-agent-c7j2c"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-etc-openvswitch\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-cni-bin\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb-hosts-file\") pod \"node-resolver-2lhxc\" (UID: \"53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb\") " pod="openshift-dns/node-resolver-2lhxc"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-etc-selinux\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-multus-cni-dir\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76444aa9-03ec-4980-97f4-dec9f17bbdc6-cni-binary-copy\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.389309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-run-ovn\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-socket-dir\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-etc-selinux\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.388962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-etc-kubernetes\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/874f202e-075e-49fd-b6c9-2729f95eb09e-konnectivity-ca\") pod \"konnectivity-agent-c7j2c\" (UID: \"874f202e-075e-49fd-b6c9-2729f95eb09e\") " pod="kube-system/konnectivity-agent-c7j2c"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-log-socket\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-tuned\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dabdc4e-dedb-449e-91eb-56201e293152-ovnkube-config\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dabdc4e-dedb-449e-91eb-56201e293152-ovnkube-script-lib\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-registration-dir\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.389978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-sysctl-d\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.390006    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6592a0ab-d648-4559-aebe-a8bbf9462e1b-tmp\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.390014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-lib-modules\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.390019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/20917dba-8a4d-4cd2-9974-5147e5894bd5-registration-dir\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.390042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-run-systemd\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.390089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20b8cf45-3b5c-46f1-adb0-ebebf097e6e6-serviceca\") pod \"node-ca-4tjx2\" (UID: \"20b8cf45-3b5c-46f1-adb0-ebebf097e6e6\") " pod="openshift-image-registry/node-ca-4tjx2"
Jul 09 23:28:04.390179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.390116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6g7v\" (UniqueName: \"kubernetes.io/projected/20b8cf45-3b5c-46f1-adb0-ebebf097e6e6-kube-api-access-x6g7v\") pod \"node-ca-4tjx2\" (UID: \"20b8cf45-3b5c-46f1-adb0-ebebf097e6e6\") " pod="openshift-image-registry/node-ca-4tjx2"
Jul 09 23:28:04.390950 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.390120    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/874f202e-075e-49fd-b6c9-2729f95eb09e-agent-certs\") pod \"konnectivity-agent-c7j2c\" (UID: \"874f202e-075e-49fd-b6c9-2729f95eb09e\") " pod="kube-system/konnectivity-agent-c7j2c"
Jul 09 23:28:04.390950 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.390136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-lib-modules\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.390950 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.390158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6592a0ab-d648-4559-aebe-a8bbf9462e1b-etc-sysctl-d\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.390950 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.390244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76444aa9-03ec-4980-97f4-dec9f17bbdc6-multus-cni-dir\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.390950 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.390410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76444aa9-03ec-4980-97f4-dec9f17bbdc6-cni-binary-copy\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.397391 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.397365    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 23:28:04.397556 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.397398    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 23:28:04.397556 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.397413    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nxdwx for pod openshift-network-diagnostics/network-check-target-hgsp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:04.397556 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.397484    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx podName:d10a13cc-9e45-49e7-a53b-e03df17e48e7 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:04.897464557 +0000 UTC m=+3.041395341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nxdwx" (UniqueName: "kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx") pod "network-check-target-hgsp8" (UID: "d10a13cc-9e45-49e7-a53b-e03df17e48e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:04.400290 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.400223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kl2\" (UniqueName: \"kubernetes.io/projected/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-kube-api-access-55kl2\") pod \"network-metrics-daemon-m5bj7\" (UID: \"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8\") " pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:04.400411 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.400358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4jcl\" (UniqueName: \"kubernetes.io/projected/20917dba-8a4d-4cd2-9974-5147e5894bd5-kube-api-access-f4jcl\") pod \"aws-ebs-csi-driver-node-kx94q\" (UID: \"20917dba-8a4d-4cd2-9974-5147e5894bd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.400671 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.400650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqffh\" (UniqueName: \"kubernetes.io/projected/6592a0ab-d648-4559-aebe-a8bbf9462e1b-kube-api-access-rqffh\") pod \"tuned-jrmhz\" (UID: \"6592a0ab-d648-4559-aebe-a8bbf9462e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.400820 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.400794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98nmb\" (UniqueName: \"kubernetes.io/projected/76444aa9-03ec-4980-97f4-dec9f17bbdc6-kube-api-access-98nmb\") pod \"multus-grfjv\" (UID: \"76444aa9-03ec-4980-97f4-dec9f17bbdc6\") " pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.401370 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.401349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9869t\" (UniqueName: \"kubernetes.io/projected/8083d0cd-a5a4-42fe-863a-e70e5b0e8a55-kube-api-access-9869t\") pod \"multus-additional-cni-plugins-p96p7\" (UID: \"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55\") " pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.490901 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.490811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdnqv\" (UniqueName: \"kubernetes.io/projected/2dabdc4e-dedb-449e-91eb-56201e293152-kube-api-access-pdnqv\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.490901 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.490858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-slash\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.490901 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.490887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-cni-netd\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.490914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/febddf31-55fc-4ec0-9ebb-019c5952772d-host-slash\") pod \"iptables-alerter-6vbbs\" (UID: \"febddf31-55fc-4ec0-9ebb-019c5952772d\") " pod="openshift-network-operator/iptables-alerter-6vbbs"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.490935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-var-lib-openvswitch\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.490959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb-tmp-dir\") pod \"node-resolver-2lhxc\" (UID: \"53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb\") " pod="openshift-dns/node-resolver-2lhxc"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.490985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-kubelet\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.490995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-cni-netd\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fe6a62a4-df2a-4d08-b134-213d033b0e28-kubelet-config\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-node-log\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-slash\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491071    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fe6a62a4-df2a-4d08-b134-213d033b0e28-kubelet-config\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-node-log\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/febddf31-55fc-4ec0-9ebb-019c5952772d-host-slash\") pod \"iptables-alerter-6vbbs\" (UID: \"febddf31-55fc-4ec0-9ebb-019c5952772d\") " pod="openshift-network-operator/iptables-alerter-6vbbs"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dabdc4e-dedb-449e-91eb-56201e293152-env-overrides\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/febddf31-55fc-4ec0-9ebb-019c5952772d-iptables-alerter-script\") pod \"iptables-alerter-6vbbs\" (UID: \"febddf31-55fc-4ec0-9ebb-019c5952772d\") " pod="openshift-network-operator/iptables-alerter-6vbbs"
Jul 09 23:28:04.491217 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-systemd-units\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fe6a62a4-df2a-4d08-b134-213d033b0e28-dbus\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-kubelet\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-var-lib-openvswitch\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-systemd-units\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dabdc4e-dedb-449e-91eb-56201e293152-ovn-node-metrics-cert\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fe6a62a4-df2a-4d08-b134-213d033b0e28-dbus\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-run-ovn-kubernetes\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrkg\" (UniqueName: \"kubernetes.io/projected/53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb-kube-api-access-fkrkg\") pod \"node-resolver-2lhxc\" (UID: \"53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb\") " pod="openshift-dns/node-resolver-2lhxc"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-etc-openvswitch\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-cni-bin\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-etc-openvswitch\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb-hosts-file\") pod \"node-resolver-2lhxc\" (UID: \"53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb\") " pod="openshift-dns/node-resolver-2lhxc"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-cni-bin\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb-tmp-dir\") pod \"node-resolver-2lhxc\" (UID: \"53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb\") " pod="openshift-dns/node-resolver-2lhxc"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb-hosts-file\") pod \"node-resolver-2lhxc\" (UID: \"53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb\") " pod="openshift-dns/node-resolver-2lhxc"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-run-ovn-kubernetes\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-run-ovn\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-run-ovn\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-log-socket\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-log-socket\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dabdc4e-dedb-449e-91eb-56201e293152-ovnkube-config\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dabdc4e-dedb-449e-91eb-56201e293152-ovnkube-script-lib\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dabdc4e-dedb-449e-91eb-56201e293152-env-overrides\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-run-systemd\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20b8cf45-3b5c-46f1-adb0-ebebf097e6e6-serviceca\") pod \"node-ca-4tjx2\" (UID: \"20b8cf45-3b5c-46f1-adb0-ebebf097e6e6\") " pod="openshift-image-registry/node-ca-4tjx2"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6g7v\" (UniqueName: \"kubernetes.io/projected/20b8cf45-3b5c-46f1-adb0-ebebf097e6e6-kube-api-access-x6g7v\") pod \"node-ca-4tjx2\" (UID: \"20b8cf45-3b5c-46f1-adb0-ebebf097e6e6\") " pod="openshift-image-registry/node-ca-4tjx2"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.491769    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/febddf31-55fc-4ec0-9ebb-019c5952772d-iptables-alerter-script\") pod \"iptables-alerter-6vbbs\" (UID: \"febddf31-55fc-4ec0-9ebb-019c5952772d\") " pod="openshift-network-operator/iptables-alerter-6vbbs"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491816    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-run-systemd\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.491828    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret podName:fe6a62a4-df2a-4d08-b134-213d033b0e28 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:04.991810392 +0000 UTC m=+3.135741192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret") pod "global-pull-secret-syncer-tmmgg" (UID: "fe6a62a4-df2a-4d08-b134-213d033b0e28") : object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-run-netns\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20b8cf45-3b5c-46f1-adb0-ebebf097e6e6-host\") pod \"node-ca-4tjx2\" (UID: \"20b8cf45-3b5c-46f1-adb0-ebebf097e6e6\") " pod="openshift-image-registry/node-ca-4tjx2"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww8vh\" (UniqueName: \"kubernetes.io/projected/febddf31-55fc-4ec0-9ebb-019c5952772d-kube-api-access-ww8vh\") pod \"iptables-alerter-6vbbs\" (UID: \"febddf31-55fc-4ec0-9ebb-019c5952772d\") " pod="openshift-network-operator/iptables-alerter-6vbbs"
Jul 09 23:28:04.492902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-run-openvswitch\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.493592 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.491966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20b8cf45-3b5c-46f1-adb0-ebebf097e6e6-host\") pod \"node-ca-4tjx2\" (UID: \"20b8cf45-3b5c-46f1-adb0-ebebf097e6e6\") " pod="openshift-image-registry/node-ca-4tjx2"
Jul 09 23:28:04.493592 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.492003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-run-openvswitch\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.493592 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.492008    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dabdc4e-dedb-449e-91eb-56201e293152-host-run-netns\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.493592 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.492058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dabdc4e-dedb-449e-91eb-56201e293152-ovnkube-config\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.493592 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.492212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dabdc4e-dedb-449e-91eb-56201e293152-ovnkube-script-lib\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.493592 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.492273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20b8cf45-3b5c-46f1-adb0-ebebf097e6e6-serviceca\") pod \"node-ca-4tjx2\" (UID: \"20b8cf45-3b5c-46f1-adb0-ebebf097e6e6\") " pod="openshift-image-registry/node-ca-4tjx2"
Jul 09 23:28:04.494069 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.494014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dabdc4e-dedb-449e-91eb-56201e293152-ovn-node-metrics-cert\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.503779 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.503744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6g7v\" (UniqueName: \"kubernetes.io/projected/20b8cf45-3b5c-46f1-adb0-ebebf097e6e6-kube-api-access-x6g7v\") pod \"node-ca-4tjx2\" (UID: \"20b8cf45-3b5c-46f1-adb0-ebebf097e6e6\") " pod="openshift-image-registry/node-ca-4tjx2"
Jul 09 23:28:04.504396 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.504371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww8vh\" (UniqueName: \"kubernetes.io/projected/febddf31-55fc-4ec0-9ebb-019c5952772d-kube-api-access-ww8vh\") pod \"iptables-alerter-6vbbs\" (UID: \"febddf31-55fc-4ec0-9ebb-019c5952772d\") " pod="openshift-network-operator/iptables-alerter-6vbbs"
Jul 09 23:28:04.504737 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.504714    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrkg\" (UniqueName: \"kubernetes.io/projected/53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb-kube-api-access-fkrkg\") pod \"node-resolver-2lhxc\" (UID: \"53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb\") " pod="openshift-dns/node-resolver-2lhxc"
Jul 09 23:28:04.504844 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.504810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdnqv\" (UniqueName: \"kubernetes.io/projected/2dabdc4e-dedb-449e-91eb-56201e293152-kube-api-access-pdnqv\") pod \"ovnkube-node-rvp85\" (UID: \"2dabdc4e-dedb-449e-91eb-56201e293152\") " pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.574929 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.574893    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p96p7"
Jul 09 23:28:04.587965 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.587939    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q"
Jul 09 23:28:04.595945 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.595923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-grfjv"
Jul 09 23:28:04.601809 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.601781    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c7j2c"
Jul 09 23:28:04.609572 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.609539    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jrmhz"
Jul 09 23:28:04.617491 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.617462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2lhxc"
Jul 09 23:28:04.626147 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.626114    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4tjx2"
Jul 09 23:28:04.635146 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.635121    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6vbbs"
Jul 09 23:28:04.641221 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.641195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:04.648362 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.648336    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 23:28:04.894990 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.894899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs\") pod \"network-metrics-daemon-m5bj7\" (UID: \"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8\") " pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:04.895165 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.895026    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:04.895165 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.895100    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs podName:1e7add64-a9cc-4acc-9102-a8c5efbbc5a8 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:05.8950761 +0000 UTC m=+4.039006900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs") pod "network-metrics-daemon-m5bj7" (UID: "1e7add64-a9cc-4acc-9102-a8c5efbbc5a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:04.995998 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.995968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdwx\" (UniqueName: \"kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx\") pod \"network-check-target-hgsp8\" (UID: \"d10a13cc-9e45-49e7-a53b-e03df17e48e7\") " pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:04.996140 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:04.996030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:04.996140 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.996122    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:04.996235 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.996138    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 23:28:04.996235 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.996158    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 23:28:04.996235 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.996171    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nxdwx for pod openshift-network-diagnostics/network-check-target-hgsp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:04.996235 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.996175    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret podName:fe6a62a4-df2a-4d08-b134-213d033b0e28 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:05.996160294 +0000 UTC m=+4.140091059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret") pod "global-pull-secret-syncer-tmmgg" (UID: "fe6a62a4-df2a-4d08-b134-213d033b0e28") : object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:04.996235 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:04.996221    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx podName:d10a13cc-9e45-49e7-a53b-e03df17e48e7 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:05.996202074 +0000 UTC m=+4.140132839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxdwx" (UniqueName: "kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx") pod "network-check-target-hgsp8" (UID: "d10a13cc-9e45-49e7-a53b-e03df17e48e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:05.314590 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:05.314485    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 23:23:04 +0000 UTC" deadline="2028-01-02 21:13:20.605025843 +0000 UTC"
Jul 09 23:28:05.314590 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:05.314531    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13005h45m15.290498389s"
Jul 09 23:28:05.369972 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:05.369937    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8083d0cd_a5a4_42fe_863a_e70e5b0e8a55.slice/crio-5baa9fca5f0bfdc553c7025783bb50849c5e1a90eacf7f2d12a6bd942c727051 WatchSource:0}: Error finding container 5baa9fca5f0bfdc553c7025783bb50849c5e1a90eacf7f2d12a6bd942c727051: Status 404 returned error can't find the container with id 5baa9fca5f0bfdc553c7025783bb50849c5e1a90eacf7f2d12a6bd942c727051
Jul 09 23:28:05.371280 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:05.371242    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebddf31_55fc_4ec0_9ebb_019c5952772d.slice/crio-04383b074eb07cfec8370cb6bc71f7f6673d286ebf9ef4a6f70bcff80736db5c WatchSource:0}: Error finding container 04383b074eb07cfec8370cb6bc71f7f6673d286ebf9ef4a6f70bcff80736db5c: Status 404 returned error can't find the container with id 04383b074eb07cfec8370cb6bc71f7f6673d286ebf9ef4a6f70bcff80736db5c
Jul 09 23:28:05.372391 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:05.372366    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dabdc4e_dedb_449e_91eb_56201e293152.slice/crio-8f1a653237450bd0a49ae44dde9537f4b533fb79b6651887239f49e1a0b72dc4 WatchSource:0}: Error finding container 8f1a653237450bd0a49ae44dde9537f4b533fb79b6651887239f49e1a0b72dc4: Status 404 returned error can't find the container with id 8f1a653237450bd0a49ae44dde9537f4b533fb79b6651887239f49e1a0b72dc4
Jul 09 23:28:05.376765 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:05.376631    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6592a0ab_d648_4559_aebe_a8bbf9462e1b.slice/crio-51aac16df98fe263ea73d5dc1523febf75eaebb1c84a31e38833f453a07baa56 WatchSource:0}: Error finding container 51aac16df98fe263ea73d5dc1523febf75eaebb1c84a31e38833f453a07baa56: Status 404 returned error can't find the container with id 51aac16df98fe263ea73d5dc1523febf75eaebb1c84a31e38833f453a07baa56
Jul 09 23:28:05.377575 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:05.377554    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874f202e_075e_49fd_b6c9_2729f95eb09e.slice/crio-f89bcb373f71cbcbe44327d1a010d730b03b0d62497803ed797193eb42b327df WatchSource:0}: Error finding container f89bcb373f71cbcbe44327d1a010d730b03b0d62497803ed797193eb42b327df: Status 404 returned error can't find the container with id f89bcb373f71cbcbe44327d1a010d730b03b0d62497803ed797193eb42b327df
Jul 09 23:28:05.378538 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:05.378336    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ac79d2_b750_4cdd_a5bb_dffe1b5c77cb.slice/crio-238b4f3176f2c88971902e25e4a912fddd1dbc29860afa9902e1520c9f3277ca WatchSource:0}: Error finding container 238b4f3176f2c88971902e25e4a912fddd1dbc29860afa9902e1520c9f3277ca: Status 404 returned error can't find the container with id 238b4f3176f2c88971902e25e4a912fddd1dbc29860afa9902e1520c9f3277ca
Jul 09 23:28:05.383658 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:05.383627    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b8cf45_3b5c_46f1_adb0_ebebf097e6e6.slice/crio-0ce4a6961aec00e0892a1417be6306aa07dfbbaa5498497dbe057018749d8c61 WatchSource:0}: Error finding container 0ce4a6961aec00e0892a1417be6306aa07dfbbaa5498497dbe057018749d8c61: Status 404 returned error can't find the container with id 0ce4a6961aec00e0892a1417be6306aa07dfbbaa5498497dbe057018749d8c61
Jul 09 23:28:05.413521 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:05.413498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:05.413633 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:05.413603    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:05.422744 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:05.422711    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2lhxc" event={"ID":"53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb","Type":"ContainerStarted","Data":"238b4f3176f2c88971902e25e4a912fddd1dbc29860afa9902e1520c9f3277ca"}
Jul 09 23:28:05.423836 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:05.423815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jrmhz" event={"ID":"6592a0ab-d648-4559-aebe-a8bbf9462e1b","Type":"ContainerStarted","Data":"51aac16df98fe263ea73d5dc1523febf75eaebb1c84a31e38833f453a07baa56"}
Jul 09 23:28:05.424905 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:05.424882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85" event={"ID":"2dabdc4e-dedb-449e-91eb-56201e293152","Type":"ContainerStarted","Data":"8f1a653237450bd0a49ae44dde9537f4b533fb79b6651887239f49e1a0b72dc4"}
Jul 09 23:28:05.426134 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:05.425973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c7j2c" event={"ID":"874f202e-075e-49fd-b6c9-2729f95eb09e","Type":"ContainerStarted","Data":"f89bcb373f71cbcbe44327d1a010d730b03b0d62497803ed797193eb42b327df"}
Jul 09 23:28:05.427042 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:05.427024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6vbbs" event={"ID":"febddf31-55fc-4ec0-9ebb-019c5952772d","Type":"ContainerStarted","Data":"04383b074eb07cfec8370cb6bc71f7f6673d286ebf9ef4a6f70bcff80736db5c"}
Jul 09 23:28:05.428221 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:05.428190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p96p7" event={"ID":"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55","Type":"ContainerStarted","Data":"5baa9fca5f0bfdc553c7025783bb50849c5e1a90eacf7f2d12a6bd942c727051"}
Jul 09 23:28:05.429536 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:05.429509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-grfjv" event={"ID":"76444aa9-03ec-4980-97f4-dec9f17bbdc6","Type":"ContainerStarted","Data":"062c4eeeefd10a4e94bcdf650df81b5545dd76bb510e1b8287a7e2d0a3947757"}
Jul 09 23:28:05.430837 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:05.430815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q" event={"ID":"20917dba-8a4d-4cd2-9974-5147e5894bd5","Type":"ContainerStarted","Data":"f69e24b06792800075481dba1b26e8c4ea5525fee58b5841e4c11935f3a92805"}
Jul 09 23:28:05.434953 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:05.434898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4tjx2" event={"ID":"20b8cf45-3b5c-46f1-adb0-ebebf097e6e6","Type":"ContainerStarted","Data":"0ce4a6961aec00e0892a1417be6306aa07dfbbaa5498497dbe057018749d8c61"}
Jul 09 23:28:05.902898 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:05.902808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs\") pod \"network-metrics-daemon-m5bj7\" (UID: \"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8\") " pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:05.903054 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:05.903025    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:05.903115 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:05.903097    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs podName:1e7add64-a9cc-4acc-9102-a8c5efbbc5a8 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:07.903074866 +0000 UTC m=+6.047005646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs") pod "network-metrics-daemon-m5bj7" (UID: "1e7add64-a9cc-4acc-9102-a8c5efbbc5a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:06.004901 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:06.004013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdwx\" (UniqueName: \"kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx\") pod \"network-check-target-hgsp8\" (UID: \"d10a13cc-9e45-49e7-a53b-e03df17e48e7\") " pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:06.004901 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:06.004122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:06.004901 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:06.004255    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:06.004901 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:06.004324    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret podName:fe6a62a4-df2a-4d08-b134-213d033b0e28 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:08.004301716 +0000 UTC m=+6.148232484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret") pod "global-pull-secret-syncer-tmmgg" (UID: "fe6a62a4-df2a-4d08-b134-213d033b0e28") : object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:06.004901 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:06.004777    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 23:28:06.004901 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:06.004797    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 23:28:06.004901 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:06.004811    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nxdwx for pod openshift-network-diagnostics/network-check-target-hgsp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:06.004901 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:06.004858    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx podName:d10a13cc-9e45-49e7-a53b-e03df17e48e7 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:08.004843219 +0000 UTC m=+6.148774003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxdwx" (UniqueName: "kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx") pod "network-check-target-hgsp8" (UID: "d10a13cc-9e45-49e7-a53b-e03df17e48e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:06.315516 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:06.315363    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 23:23:04 +0000 UTC" deadline="2028-03-15 20:21:44.525040257 +0000 UTC"
Jul 09 23:28:06.315516 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:06.315427    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14756h53m38.209617792s"
Jul 09 23:28:06.416328 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:06.416289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:06.416524 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:06.416480    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:06.416980 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:06.416958    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:06.417092 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:06.417067    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:06.447433 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:06.447391    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal" event={"ID":"3203ef60a382e0f7b0ec847ea9d1892c","Type":"ContainerStarted","Data":"36bc58a100282b9603fbdd0b6369844c02afd3dbcec8e4a106d6b2afd37f69f9"}
Jul 09 23:28:06.457378 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:06.457342    2580 generic.go:356] "Generic (PLEG): container finished" podID="2584345ba2d507ebd069c4f5d2f9d4a9" containerID="3fb68dca5683b0d0c1e75063cbeea440a89bcceeb6f68d4f58623debcc71f69c" exitCode=0
Jul 09 23:28:06.457548 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:06.457404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" event={"ID":"2584345ba2d507ebd069c4f5d2f9d4a9","Type":"ContainerDied","Data":"3fb68dca5683b0d0c1e75063cbeea440a89bcceeb6f68d4f58623debcc71f69c"}
Jul 09 23:28:06.477809 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:06.477751    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-8.ec2.internal" podStartSLOduration=2.47770331 podStartE2EDuration="2.47770331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:06.461936172 +0000 UTC m=+4.605866963" watchObservedRunningTime="2026-07-09 23:28:06.47770331 +0000 UTC m=+4.621634102"
Jul 09 23:28:07.413801 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:07.413266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:07.413801 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:07.413422    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:07.477480 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:07.477442    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" event={"ID":"2584345ba2d507ebd069c4f5d2f9d4a9","Type":"ContainerStarted","Data":"a019f086a36dae1b49c655e520371d6e8a092d5a44bb79b727ba509721d39c07"}
Jul 09 23:28:07.921191 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:07.921151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs\") pod \"network-metrics-daemon-m5bj7\" (UID: \"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8\") " pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:07.921378 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:07.921364    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:07.921468 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:07.921431    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs podName:1e7add64-a9cc-4acc-9102-a8c5efbbc5a8 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:11.921410771 +0000 UTC m=+10.065341545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs") pod "network-metrics-daemon-m5bj7" (UID: "1e7add64-a9cc-4acc-9102-a8c5efbbc5a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:08.021918 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:08.021878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:08.022125 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:08.021949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdwx\" (UniqueName: \"kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx\") pod \"network-check-target-hgsp8\" (UID: \"d10a13cc-9e45-49e7-a53b-e03df17e48e7\") " pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:08.022194 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:08.022151    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 23:28:08.022194 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:08.022173    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 23:28:08.022194 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:08.022188    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nxdwx for pod openshift-network-diagnostics/network-check-target-hgsp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:08.022354 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:08.022252    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx podName:d10a13cc-9e45-49e7-a53b-e03df17e48e7 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:12.022230986 +0000 UTC m=+10.166161755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxdwx" (UniqueName: "kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx") pod "network-check-target-hgsp8" (UID: "d10a13cc-9e45-49e7-a53b-e03df17e48e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:08.022937 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:08.022720    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:08.022937 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:08.022796    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret podName:fe6a62a4-df2a-4d08-b134-213d033b0e28 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:12.02277811 +0000 UTC m=+10.166708877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret") pod "global-pull-secret-syncer-tmmgg" (UID: "fe6a62a4-df2a-4d08-b134-213d033b0e28") : object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:08.414927 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:08.414842    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:08.415393 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:08.415031    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:08.415795 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:08.415584    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:08.415795 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:08.415708    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:09.413155 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:09.413116    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:09.413348 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:09.413268    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:10.413327 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:10.413293    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:10.413790 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:10.413424    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:10.413790 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:10.413467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:10.413790 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:10.413568    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:11.413552 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:11.413520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:11.414031 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:11.413681    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:11.962243 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:11.962207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs\") pod \"network-metrics-daemon-m5bj7\" (UID: \"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8\") " pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:11.962422 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:11.962356    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:11.962422 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:11.962417    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs podName:1e7add64-a9cc-4acc-9102-a8c5efbbc5a8 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:19.962397031 +0000 UTC m=+18.106327814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs") pod "network-metrics-daemon-m5bj7" (UID: "1e7add64-a9cc-4acc-9102-a8c5efbbc5a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:12.063610 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:12.063572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdwx\" (UniqueName: \"kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx\") pod \"network-check-target-hgsp8\" (UID: \"d10a13cc-9e45-49e7-a53b-e03df17e48e7\") " pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:12.063810 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:12.063692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:12.063810 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:12.063784    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 23:28:12.063810 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:12.063799    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:12.063810 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:12.063809    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 23:28:12.064016 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:12.063825    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nxdwx for pod openshift-network-diagnostics/network-check-target-hgsp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:12.064016 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:12.063858    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret podName:fe6a62a4-df2a-4d08-b134-213d033b0e28 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:20.063839345 +0000 UTC m=+18.207770128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret") pod "global-pull-secret-syncer-tmmgg" (UID: "fe6a62a4-df2a-4d08-b134-213d033b0e28") : object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:12.064016 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:12.063880    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx podName:d10a13cc-9e45-49e7-a53b-e03df17e48e7 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:20.063868997 +0000 UTC m=+18.207799768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxdwx" (UniqueName: "kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx") pod "network-check-target-hgsp8" (UID: "d10a13cc-9e45-49e7-a53b-e03df17e48e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:12.414743 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:12.414113    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:12.414743 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:12.414223    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:12.414743 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:12.414273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:12.414743 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:12.414344    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:13.414059 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:13.413311    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:13.414059 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:13.413466    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:14.413857 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:14.413732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:14.414296 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:14.413871    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:14.414296 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:14.413732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:14.414296 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:14.414041    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:15.413202 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:15.413166    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:15.413383 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:15.413317    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:16.413790 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:16.413721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:16.413790 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:16.413779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:16.414223 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:16.413863    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:16.414223 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:16.414018    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:17.413423 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:17.413386    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:17.413579 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:17.413530    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:18.413146 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:18.413110    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:18.413606 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:18.413113    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:18.413606 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:18.413240    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:18.413606 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:18.413322    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:19.413368 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:19.413334    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:19.413831 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:19.413459    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:20.021347 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:20.021306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs\") pod \"network-metrics-daemon-m5bj7\" (UID: \"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8\") " pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:20.021535 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:20.021481    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:20.021579 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:20.021563    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs podName:1e7add64-a9cc-4acc-9102-a8c5efbbc5a8 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:36.021542088 +0000 UTC m=+34.165472858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs") pod "network-metrics-daemon-m5bj7" (UID: "1e7add64-a9cc-4acc-9102-a8c5efbbc5a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:20.122287 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:20.122246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:20.122476 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:20.122313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdwx\" (UniqueName: \"kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx\") pod \"network-check-target-hgsp8\" (UID: \"d10a13cc-9e45-49e7-a53b-e03df17e48e7\") " pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:20.122476 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:20.122430    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:20.122476 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:20.122461    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 23:28:20.122609 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:20.122482    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 23:28:20.122609 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:20.122496    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nxdwx for pod openshift-network-diagnostics/network-check-target-hgsp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:20.122609 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:20.122510    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret podName:fe6a62a4-df2a-4d08-b134-213d033b0e28 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:36.122487174 +0000 UTC m=+34.266417961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret") pod "global-pull-secret-syncer-tmmgg" (UID: "fe6a62a4-df2a-4d08-b134-213d033b0e28") : object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:20.122609 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:20.122534    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx podName:d10a13cc-9e45-49e7-a53b-e03df17e48e7 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:36.122523585 +0000 UTC m=+34.266454353 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxdwx" (UniqueName: "kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx") pod "network-check-target-hgsp8" (UID: "d10a13cc-9e45-49e7-a53b-e03df17e48e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:20.413473 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:20.413391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:20.413881 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:20.413392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:20.413881 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:20.413532    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:20.413881 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:20.413568    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:21.413703 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:21.413665    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:21.414164 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:21.413798    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:22.413528 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:22.413493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:22.413718 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:22.413595    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:22.413718 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:22.413661    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:22.414134 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:22.413761    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:23.413695 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:23.413661    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:23.413901 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:23.413788    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:24.413721 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:24.413495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:24.413721 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:24.413616    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:24.414502 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:24.414359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:24.414502 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:24.414464    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:24.516534 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:24.516312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c7j2c" event={"ID":"874f202e-075e-49fd-b6c9-2729f95eb09e","Type":"ContainerStarted","Data":"d4e01b5ff4971c410d3d281b2439169e9f4657309f42e0a1d675d4c9a24f6ef6"}
Jul 09 23:28:24.519116 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:24.519084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p96p7" event={"ID":"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55","Type":"ContainerStarted","Data":"dad8536b22f4faa6dbe91cedd04da571704ad604791ec604bf67c7c8dbb5a546"}
Jul 09 23:28:24.525833 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:24.525766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-grfjv" event={"ID":"76444aa9-03ec-4980-97f4-dec9f17bbdc6","Type":"ContainerStarted","Data":"1e0ecd75b6e8b7cfeaf0b798ea45737bdab12c1b243af823bbe36ac885a054be"}
Jul 09 23:28:24.534016 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:24.533148    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c7j2c" podStartSLOduration=6.855412174 podStartE2EDuration="21.533128918s" totalImagesPullingTime="14.677716744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:24.532063435 +0000 UTC m=+22.675994222" watchObservedRunningTime="2026-07-09 23:28:24.533128918 +0000 UTC m=+22.677059706"
Jul 09 23:28:24.534016 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:24.533458    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-8.ec2.internal" podStartSLOduration=20.533447224 podStartE2EDuration="20.533447224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:07.492391428 +0000 UTC m=+5.636322218" watchObservedRunningTime="2026-07-09 23:28:24.533447224 +0000 UTC m=+22.677378031"
Jul 09 23:28:24.537302 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:24.537133    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jrmhz" event={"ID":"6592a0ab-d648-4559-aebe-a8bbf9462e1b","Type":"ContainerStarted","Data":"ae5e630ea5d20401f4fbf85ca1874b886b6b5dfa5a38d576e7e76344c13a4f81"}
Jul 09 23:28:24.568948 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:24.568895    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-grfjv" podStartSLOduration=2.549797479 podStartE2EDuration="21.56887475s" totalImagesPullingTime="19.019077271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:24.568826522 +0000 UTC m=+22.712757331" watchObservedRunningTime="2026-07-09 23:28:24.56887475 +0000 UTC m=+22.712805536"
Jul 09 23:28:24.598806 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:24.598243    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jrmhz" podStartSLOduration=2.639812605 podStartE2EDuration="21.59822222s" totalImagesPullingTime="18.958409615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:24.597667099 +0000 UTC m=+22.741597877" watchObservedRunningTime="2026-07-09 23:28:24.59822222 +0000 UTC m=+22.742153007"
Jul 09 23:28:24.599510 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:24.599458    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4tjx2" podStartSLOduration=2.648610995 podStartE2EDuration="21.599439275s" totalImagesPullingTime="18.95082828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:24.582382954 +0000 UTC m=+22.726313742" watchObservedRunningTime="2026-07-09 23:28:24.599439275 +0000 UTC m=+22.743370062"
Jul 09 23:28:24.614054 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:24.613895    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2lhxc" podStartSLOduration=2.659176681 podStartE2EDuration="21.613878458s" totalImagesPullingTime="18.954701777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:24.613453255 +0000 UTC m=+22.757384044" watchObservedRunningTime="2026-07-09 23:28:24.613878458 +0000 UTC m=+22.757809243"
Jul 09 23:28:25.412764 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:25.412731    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:25.412911 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:25.412838    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:25.541132 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:25.541100    2580 generic.go:356] "Generic (PLEG): container finished" podID="8083d0cd-a5a4-42fe-863a-e70e5b0e8a55" containerID="dad8536b22f4faa6dbe91cedd04da571704ad604791ec604bf67c7c8dbb5a546" exitCode=0
Jul 09 23:28:25.541945 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:25.541173    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p96p7" event={"ID":"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55","Type":"ContainerDied","Data":"dad8536b22f4faa6dbe91cedd04da571704ad604791ec604bf67c7c8dbb5a546"}
Jul 09 23:28:25.542630 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:25.542608    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q" event={"ID":"20917dba-8a4d-4cd2-9974-5147e5894bd5","Type":"ContainerStarted","Data":"f490cc7a8983283239241c5f7ce3b10224bd6bbd32e6b15a0b87a1e28ae58718"}
Jul 09 23:28:25.543924 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:25.543898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4tjx2" event={"ID":"20b8cf45-3b5c-46f1-adb0-ebebf097e6e6","Type":"ContainerStarted","Data":"05736bbfc66eac7db439a8f9fb2b8f688ff432c38bc73b931dc9332764706729"}
Jul 09 23:28:25.545295 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:25.545266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2lhxc" event={"ID":"53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb","Type":"ContainerStarted","Data":"75c5ab019162d0f26d875304f8030869d83262755e13c660a9960b9e0980c45e"}
Jul 09 23:28:25.548130 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:25.548107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85" event={"ID":"2dabdc4e-dedb-449e-91eb-56201e293152","Type":"ContainerStarted","Data":"018d9576a1780e8a3d5f84d6dcb6e8cadee6088af64fe7f91f92d54f60b94596"}
Jul 09 23:28:25.548215 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:25.548174    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85" event={"ID":"2dabdc4e-dedb-449e-91eb-56201e293152","Type":"ContainerStarted","Data":"291f9242b77f1703342cf2b9a1bcf4860f16205c2b65b5859dc40418803ec58c"}
Jul 09 23:28:25.548215 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:25.548211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85" event={"ID":"2dabdc4e-dedb-449e-91eb-56201e293152","Type":"ContainerStarted","Data":"8c987b71d60e6db846a952794fd7f26d4df54769f26641244501a7dd0d5e29e3"}
Jul 09 23:28:25.548294 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:25.548254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85" event={"ID":"2dabdc4e-dedb-449e-91eb-56201e293152","Type":"ContainerStarted","Data":"87a781d927f63de9551e477138f23cf2caeed4ff5d10197d4b1ae952cd7e9627"}
Jul 09 23:28:25.548338 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:25.548303    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85" event={"ID":"2dabdc4e-dedb-449e-91eb-56201e293152","Type":"ContainerStarted","Data":"1b27af055ced0cd3644e03ac998a987b347a3882cf7c03e1e82d9d98b4d5c829"}
Jul 09 23:28:25.548370 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:25.548349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85" event={"ID":"2dabdc4e-dedb-449e-91eb-56201e293152","Type":"ContainerStarted","Data":"9dc34f53ea7a5d0397e43edb98828bf053dfc71e8921aba58fbbdf43b7545128"}
Jul 09 23:28:25.741666 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:25.741632    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 23:28:26.341923 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:26.341795    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T23:28:25.741648159Z","UUID":"99afa41c-78da-4aa8-a15b-7a57ac0aeb8b","Handler":null,"Name":"","Endpoint":""}
Jul 09 23:28:26.343907 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:26.343882    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 23:28:26.344025 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:26.343927    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 23:28:26.413441 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:26.413404    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:26.413601 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:26.413405    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:26.413601 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:26.413535    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:26.413601 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:26.413578    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:26.552589 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:26.552552    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6vbbs" event={"ID":"febddf31-55fc-4ec0-9ebb-019c5952772d","Type":"ContainerStarted","Data":"b2d0e90a43b87f95cea4741180a6dd7bd50cbfe193f219cd246eea9af5cd4b99"}
Jul 09 23:28:26.554897 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:26.554868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q" event={"ID":"20917dba-8a4d-4cd2-9974-5147e5894bd5","Type":"ContainerStarted","Data":"1da4976f9b5f6d56ce1fbaa5f8bc05a6171cb204c3f260da3a222561c45a2b1f"}
Jul 09 23:28:26.567924 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:26.567873    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6vbbs" podStartSLOduration=4.60543774 podStartE2EDuration="23.567855138s" totalImagesPullingTime="18.962417398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:26.567760322 +0000 UTC m=+24.711691109" watchObservedRunningTime="2026-07-09 23:28:26.567855138 +0000 UTC m=+24.711785923"
Jul 09 23:28:27.413737 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:27.413703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:27.413909 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:27.413855    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:27.559913 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:27.559871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q" event={"ID":"20917dba-8a4d-4cd2-9974-5147e5894bd5","Type":"ContainerStarted","Data":"c99479bde792e370383494ca318be519ed70ed8ac35e792fbc0254515b5ded8d"}
Jul 09 23:28:27.563473 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:27.563439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85" event={"ID":"2dabdc4e-dedb-449e-91eb-56201e293152","Type":"ContainerStarted","Data":"0d3ddb2ae18de59fe6585866fffad53d79b45e879c7ab39db3cf1c168a8be7eb"}
Jul 09 23:28:27.578133 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:27.578089    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kx94q" podStartSLOduration=3.4685464 podStartE2EDuration="24.578074635s" totalImagesPullingTime="21.109528235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:28:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:27.577605871 +0000 UTC m=+25.721536660" watchObservedRunningTime="2026-07-09 23:28:27.578074635 +0000 UTC m=+25.722005450"
Jul 09 23:28:28.413624 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:28.413447    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:28.413871 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:28.413517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:28.413871 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:28.413728    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:28.413871 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:28.413791    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:28.935992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:28.935959    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c7j2c"
Jul 09 23:28:28.936666 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:28.936635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c7j2c"
Jul 09 23:28:29.413324 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:29.413289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:29.413491 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:29.413417    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:29.567948 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:29.567915    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c7j2c"
Jul 09 23:28:29.568508 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:29.568485    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c7j2c"
Jul 09 23:28:30.413596 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:30.413571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:30.414143 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:30.413571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:30.414143 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:30.413724    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:30.414143 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:30.413796    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:30.573039 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:30.572894    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85" event={"ID":"2dabdc4e-dedb-449e-91eb-56201e293152","Type":"ContainerStarted","Data":"71e3834f8553f4d4ac18503c8dc4730a3b0704b0fbaecb0d405abccc71a59395"}
Jul 09 23:28:30.606963 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:30.606906    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85" podStartSLOduration=8.549772247 podStartE2EDuration="27.606884652s" totalImagesPullingTime="19.057112405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:30.606277754 +0000 UTC m=+28.750208543" watchObservedRunningTime="2026-07-09 23:28:30.606884652 +0000 UTC m=+28.750815439"
Jul 09 23:28:31.413104 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:31.413070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:31.413292 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:31.413188    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:31.576134 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:31.576103    2580 generic.go:356] "Generic (PLEG): container finished" podID="8083d0cd-a5a4-42fe-863a-e70e5b0e8a55" containerID="7b53544d82c71bbdbd1a952031ba0c66b30150a2425c65aa57f86e1f4442ea79" exitCode=0
Jul 09 23:28:31.576778 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:31.576181    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p96p7" event={"ID":"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55","Type":"ContainerDied","Data":"7b53544d82c71bbdbd1a952031ba0c66b30150a2425c65aa57f86e1f4442ea79"}
Jul 09 23:28:31.576838 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:31.576785    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:31.577348 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:31.576856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:31.577348 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:31.576904    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:31.592265 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:31.592240    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:31.592427 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:31.592413    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:28:32.415377 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:32.415147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:32.415377 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:32.415256    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:32.415711 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:32.415685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:32.415831 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:32.415810    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:32.422199 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:32.422168    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m5bj7"]
Jul 09 23:28:32.422357 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:32.422288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:32.422424 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:32.422402    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:32.422933 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:32.422911    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tmmgg"]
Jul 09 23:28:32.424396 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:32.424373    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hgsp8"]
Jul 09 23:28:32.581179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:32.580929    2580 generic.go:356] "Generic (PLEG): container finished" podID="8083d0cd-a5a4-42fe-863a-e70e5b0e8a55" containerID="f2832c03ad31eb609566474458c5b635386ce504d8c695b5d427e80eeffb542e" exitCode=0
Jul 09 23:28:32.581594 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:32.581246    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:32.581594 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:32.581002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p96p7" event={"ID":"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55","Type":"ContainerDied","Data":"f2832c03ad31eb609566474458c5b635386ce504d8c695b5d427e80eeffb542e"}
Jul 09 23:28:32.581594 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:32.581347    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:32.581594 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:32.581365    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:32.581594 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:32.581446    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:34.413179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:34.413142    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:34.413179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:34.413141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:34.414140 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:34.413290    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:34.414140 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:34.413339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:34.414140 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:34.413417    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:34.414140 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:34.413493    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:34.588202 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:34.588164    2580 generic.go:356] "Generic (PLEG): container finished" podID="8083d0cd-a5a4-42fe-863a-e70e5b0e8a55" containerID="5de4d833a855f43b1b7127e102ec16c261278c42f8ea7d54b5c033afa4bdc997" exitCode=0
Jul 09 23:28:34.588388 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:34.588233    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p96p7" event={"ID":"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55","Type":"ContainerDied","Data":"5de4d833a855f43b1b7127e102ec16c261278c42f8ea7d54b5c033afa4bdc997"}
Jul 09 23:28:36.042302 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:36.042268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs\") pod \"network-metrics-daemon-m5bj7\" (UID: \"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8\") " pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:36.042815 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:36.042418    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:36.042815 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:36.042477    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs podName:1e7add64-a9cc-4acc-9102-a8c5efbbc5a8 nodeName:}" failed. No retries permitted until 2026-07-09 23:29:08.042460868 +0000 UTC m=+66.186391636 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs") pod "network-metrics-daemon-m5bj7" (UID: "1e7add64-a9cc-4acc-9102-a8c5efbbc5a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 23:28:36.143357 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:36.143322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:36.143520 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:36.143377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdwx\" (UniqueName: \"kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx\") pod \"network-check-target-hgsp8\" (UID: \"d10a13cc-9e45-49e7-a53b-e03df17e48e7\") " pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:36.143520 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:36.143473    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:36.143520 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:36.143493    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 23:28:36.143520 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:36.143507    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 23:28:36.143520 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:36.143518    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nxdwx for pod openshift-network-diagnostics/network-check-target-hgsp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:36.143735 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:36.143542    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret podName:fe6a62a4-df2a-4d08-b134-213d033b0e28 nodeName:}" failed. No retries permitted until 2026-07-09 23:29:08.143524311 +0000 UTC m=+66.287455096 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret") pod "global-pull-secret-syncer-tmmgg" (UID: "fe6a62a4-df2a-4d08-b134-213d033b0e28") : object "kube-system"/"original-pull-secret" not registered
Jul 09 23:28:36.143735 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:36.143559    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx podName:d10a13cc-9e45-49e7-a53b-e03df17e48e7 nodeName:}" failed. No retries permitted until 2026-07-09 23:29:08.143552193 +0000 UTC m=+66.287482962 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxdwx" (UniqueName: "kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx") pod "network-check-target-hgsp8" (UID: "d10a13cc-9e45-49e7-a53b-e03df17e48e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 23:28:36.412844 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:36.412808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:36.412844 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:36.412828    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:36.413091 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:36.412808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:36.413091 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:36.412935    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmmgg" podUID="fe6a62a4-df2a-4d08-b134-213d033b0e28"
Jul 09 23:28:36.413091 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:36.413013    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hgsp8" podUID="d10a13cc-9e45-49e7-a53b-e03df17e48e7"
Jul 09 23:28:36.413091 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:36.413073    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5bj7" podUID="1e7add64-a9cc-4acc-9102-a8c5efbbc5a8"
Jul 09 23:28:38.064967 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.064934    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-8.ec2.internal" event="NodeReady"
Jul 09 23:28:38.065402 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.065068    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 23:28:38.105181 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.105095    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xll26"]
Jul 09 23:28:38.107738 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.107712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:38.108492 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.108471    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c6ztd"]
Jul 09 23:28:38.110290 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.110267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c6ztd"
Jul 09 23:28:38.110290 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.110284    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 23:28:38.110455 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.110309    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8548g\""
Jul 09 23:28:38.110455 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.110289    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 23:28:38.112627 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.112606    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 23:28:38.112768 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.112676    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4bz2v\""
Jul 09 23:28:38.112768 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.112724    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 23:28:38.112924 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.112901    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 23:28:38.118478 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.118440    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xll26"]
Jul 09 23:28:38.120568 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.120542    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c6ztd"]
Jul 09 23:28:38.128140 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.128117    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7b2ch"]
Jul 09 23:28:38.130294 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.130176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.132708 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.132686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 23:28:38.132849 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.132830    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 23:28:38.132949 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.132867    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qktl8\""
Jul 09 23:28:38.133011 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.132984    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 23:28:38.133093 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.133067    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 23:28:38.140589 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.140572    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7b2ch"]
Jul 09 23:28:38.256031 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.255998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:38.256031 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.256040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15f14655-5e80-48b0-a2d4-81a4ddadaec6-cert\") pod \"ingress-canary-c6ztd\" (UID: \"15f14655-5e80-48b0-a2d4-81a4ddadaec6\") " pod="openshift-ingress-canary/ingress-canary-c6ztd"
Jul 09 23:28:38.256272 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.256085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9160d80-c17d-491e-b077-5917e5d6be05-tmp-dir\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:38.256272 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.256162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9160d80-c17d-491e-b077-5917e5d6be05-config-volume\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:38.256272 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.256196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.256272 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.256231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-data-volume\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.256272 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.256251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v4nh\" (UniqueName: \"kubernetes.io/projected/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-api-access-9v4nh\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.256272 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.256272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5bmj\" (UniqueName: \"kubernetes.io/projected/b9160d80-c17d-491e-b077-5917e5d6be05-kube-api-access-f5bmj\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:38.256483 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.256292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88v5\" (UniqueName: \"kubernetes.io/projected/15f14655-5e80-48b0-a2d4-81a4ddadaec6-kube-api-access-b88v5\") pod \"ingress-canary-c6ztd\" (UID: \"15f14655-5e80-48b0-a2d4-81a4ddadaec6\") " pod="openshift-ingress-canary/ingress-canary-c6ztd"
Jul 09 23:28:38.256483 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.256411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-crio-socket\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.256483 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.256430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.326733 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.326702    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2lhxc_53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb/dns-node-resolver/0.log"
Jul 09 23:28:38.357141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9160d80-c17d-491e-b077-5917e5d6be05-config-volume\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:38.357141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.357141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-data-volume\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.357382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v4nh\" (UniqueName: \"kubernetes.io/projected/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-api-access-9v4nh\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.357382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5bmj\" (UniqueName: \"kubernetes.io/projected/b9160d80-c17d-491e-b077-5917e5d6be05-kube-api-access-f5bmj\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:38.357382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88v5\" (UniqueName: \"kubernetes.io/projected/15f14655-5e80-48b0-a2d4-81a4ddadaec6-kube-api-access-b88v5\") pod \"ingress-canary-c6ztd\" (UID: \"15f14655-5e80-48b0-a2d4-81a4ddadaec6\") " pod="openshift-ingress-canary/ingress-canary-c6ztd"
Jul 09 23:28:38.357382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-crio-socket\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.357382 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:38.357240    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 23:28:38.357382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.357382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:38.357382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15f14655-5e80-48b0-a2d4-81a4ddadaec6-cert\") pod \"ingress-canary-c6ztd\" (UID: \"15f14655-5e80-48b0-a2d4-81a4ddadaec6\") " pod="openshift-ingress-canary/ingress-canary-c6ztd"
Jul 09 23:28:38.357382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9160d80-c17d-491e-b077-5917e5d6be05-tmp-dir\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:38.357882 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:38.357598    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 23:28:38.357882 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:38.357672    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm podName:c4ab2689-0ddb-46e2-95b9-93710ee87dc4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:38.857640112 +0000 UTC m=+37.001570893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7b2ch" (UID: "c4ab2689-0ddb-46e2-95b9-93710ee87dc4") : configmap "kube-rbac-proxy" not found
Jul 09 23:28:38.357882 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9160d80-c17d-491e-b077-5917e5d6be05-tmp-dir\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:38.357882 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-data-volume\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.357882 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:38.357710    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls podName:c4ab2689-0ddb-46e2-95b9-93710ee87dc4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:38.857683086 +0000 UTC m=+37.001613865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7b2ch" (UID: "c4ab2689-0ddb-46e2-95b9-93710ee87dc4") : secret "insights-runtime-extractor-tls" not found
Jul 09 23:28:38.357882 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:38.357792    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 23:28:38.357882 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:38.357841    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls podName:b9160d80-c17d-491e-b077-5917e5d6be05 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:38.857828613 +0000 UTC m=+37.001759392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls") pod "dns-default-xll26" (UID: "b9160d80-c17d-491e-b077-5917e5d6be05") : secret "dns-default-metrics-tls" not found
Jul 09 23:28:38.357882 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-crio-socket\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.358200 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.357896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9160d80-c17d-491e-b077-5917e5d6be05-config-volume\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:38.362208 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.362174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15f14655-5e80-48b0-a2d4-81a4ddadaec6-cert\") pod \"ingress-canary-c6ztd\" (UID: \"15f14655-5e80-48b0-a2d4-81a4ddadaec6\") " pod="openshift-ingress-canary/ingress-canary-c6ztd"
Jul 09 23:28:38.366205 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.366169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v4nh\" (UniqueName: \"kubernetes.io/projected/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-api-access-9v4nh\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.366205 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.366190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88v5\" (UniqueName: \"kubernetes.io/projected/15f14655-5e80-48b0-a2d4-81a4ddadaec6-kube-api-access-b88v5\") pod \"ingress-canary-c6ztd\" (UID: \"15f14655-5e80-48b0-a2d4-81a4ddadaec6\") " pod="openshift-ingress-canary/ingress-canary-c6ztd"
Jul 09 23:28:38.366409 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.366387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5bmj\" (UniqueName: \"kubernetes.io/projected/b9160d80-c17d-491e-b077-5917e5d6be05-kube-api-access-f5bmj\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:38.413498 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.413458    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:28:38.413699 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.413473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:28:38.413699 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.413601    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:28:38.416436 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.416298    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 23:28:38.416563 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.416479    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r9l9m\""
Jul 09 23:28:38.416563 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.416496    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8rpsb\""
Jul 09 23:28:38.416563 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.416528    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 23:28:38.417001 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.416798    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 23:28:38.417001 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.416806    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 23:28:38.426832 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.426809    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c6ztd"
Jul 09 23:28:38.623717 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.623683    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c6ztd"]
Jul 09 23:28:38.629346 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:38.629313    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f14655_5e80_48b0_a2d4_81a4ddadaec6.slice/crio-ef226ec4d96b5e90750443ef94a17198f4b271dcd3ba78193a73bc5ce28ec304 WatchSource:0}: Error finding container ef226ec4d96b5e90750443ef94a17198f4b271dcd3ba78193a73bc5ce28ec304: Status 404 returned error can't find the container with id ef226ec4d96b5e90750443ef94a17198f4b271dcd3ba78193a73bc5ce28ec304
Jul 09 23:28:38.862239 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.862134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.862386 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:38.862263    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 23:28:38.862386 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.862300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:38.862386 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:38.862324    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm podName:c4ab2689-0ddb-46e2-95b9-93710ee87dc4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:39.862309844 +0000 UTC m=+38.006240609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7b2ch" (UID: "c4ab2689-0ddb-46e2-95b9-93710ee87dc4") : configmap "kube-rbac-proxy" not found
Jul 09 23:28:38.862386 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:38.862384    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 23:28:38.862597 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:38.862382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:38.862597 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:38.862423    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls podName:b9160d80-c17d-491e-b077-5917e5d6be05 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:39.862412506 +0000 UTC m=+38.006343271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls") pod "dns-default-xll26" (UID: "b9160d80-c17d-491e-b077-5917e5d6be05") : secret "dns-default-metrics-tls" not found
Jul 09 23:28:38.862597 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:38.862470    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 23:28:38.862597 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:38.862505    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls podName:c4ab2689-0ddb-46e2-95b9-93710ee87dc4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:39.862494085 +0000 UTC m=+38.006424852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7b2ch" (UID: "c4ab2689-0ddb-46e2-95b9-93710ee87dc4") : secret "insights-runtime-extractor-tls" not found
Jul 09 23:28:39.312153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.312120    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4tjx2_20b8cf45-3b5c-46f1-adb0-ebebf097e6e6/node-ca/0.log"
Jul 09 23:28:39.341838 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.341803    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-b4k4b"]
Jul 09 23:28:39.343761 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.343738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b"
Jul 09 23:28:39.346440 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.346418    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 09 23:28:39.346558 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.346481    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 09 23:28:39.346558 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.346511    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-55mh2\""
Jul 09 23:28:39.346809 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.346418    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 09 23:28:39.347465 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.347450    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 09 23:28:39.353691 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.353645    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-b4k4b"]
Jul 09 23:28:39.467044 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.467012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c48db8d5-e73a-4c05-a0a2-c2dbf7402913-signing-key\") pod \"service-ca-79f68b9f4f-b4k4b\" (UID: \"c48db8d5-e73a-4c05-a0a2-c2dbf7402913\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b"
Jul 09 23:28:39.467044 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.467046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c48db8d5-e73a-4c05-a0a2-c2dbf7402913-signing-cabundle\") pod \"service-ca-79f68b9f4f-b4k4b\" (UID: \"c48db8d5-e73a-4c05-a0a2-c2dbf7402913\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b"
Jul 09 23:28:39.467305 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.467080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmggd\" (UniqueName: \"kubernetes.io/projected/c48db8d5-e73a-4c05-a0a2-c2dbf7402913-kube-api-access-dmggd\") pod \"service-ca-79f68b9f4f-b4k4b\" (UID: \"c48db8d5-e73a-4c05-a0a2-c2dbf7402913\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b"
Jul 09 23:28:39.567929 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.567843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c48db8d5-e73a-4c05-a0a2-c2dbf7402913-signing-key\") pod \"service-ca-79f68b9f4f-b4k4b\" (UID: \"c48db8d5-e73a-4c05-a0a2-c2dbf7402913\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b"
Jul 09 23:28:39.567929 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.567884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c48db8d5-e73a-4c05-a0a2-c2dbf7402913-signing-cabundle\") pod \"service-ca-79f68b9f4f-b4k4b\" (UID: \"c48db8d5-e73a-4c05-a0a2-c2dbf7402913\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b"
Jul 09 23:28:39.567929 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.567930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmggd\" (UniqueName: \"kubernetes.io/projected/c48db8d5-e73a-4c05-a0a2-c2dbf7402913-kube-api-access-dmggd\") pod \"service-ca-79f68b9f4f-b4k4b\" (UID: \"c48db8d5-e73a-4c05-a0a2-c2dbf7402913\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b"
Jul 09 23:28:39.568755 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.568726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c48db8d5-e73a-4c05-a0a2-c2dbf7402913-signing-cabundle\") pod \"service-ca-79f68b9f4f-b4k4b\" (UID: \"c48db8d5-e73a-4c05-a0a2-c2dbf7402913\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b"
Jul 09 23:28:39.571866 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.571843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c48db8d5-e73a-4c05-a0a2-c2dbf7402913-signing-key\") pod \"service-ca-79f68b9f4f-b4k4b\" (UID: \"c48db8d5-e73a-4c05-a0a2-c2dbf7402913\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b"
Jul 09 23:28:39.576867 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.576844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmggd\" (UniqueName: \"kubernetes.io/projected/c48db8d5-e73a-4c05-a0a2-c2dbf7402913-kube-api-access-dmggd\") pod \"service-ca-79f68b9f4f-b4k4b\" (UID: \"c48db8d5-e73a-4c05-a0a2-c2dbf7402913\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b"
Jul 09 23:28:39.602036 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.601993    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c6ztd" event={"ID":"15f14655-5e80-48b0-a2d4-81a4ddadaec6","Type":"ContainerStarted","Data":"ef226ec4d96b5e90750443ef94a17198f4b271dcd3ba78193a73bc5ce28ec304"}
Jul 09 23:28:39.605309 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.605279    2580 generic.go:356] "Generic (PLEG): container finished" podID="8083d0cd-a5a4-42fe-863a-e70e5b0e8a55" containerID="1ffd8c669ef57bad3b138ae2b7917aac05f4eedd4541defeba25ea027dfdbf87" exitCode=0
Jul 09 23:28:39.605463 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.605329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p96p7" event={"ID":"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55","Type":"ContainerDied","Data":"1ffd8c669ef57bad3b138ae2b7917aac05f4eedd4541defeba25ea027dfdbf87"}
Jul 09 23:28:39.655235 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.655201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b"
Jul 09 23:28:39.791792 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.791504    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-b4k4b"]
Jul 09 23:28:39.795447 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:39.795416    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48db8d5_e73a_4c05_a0a2_c2dbf7402913.slice/crio-48d8dc8be93444a88b43ec968e756c11849faf67c98847f1aa3b8cbe2a89b1b4 WatchSource:0}: Error finding container 48d8dc8be93444a88b43ec968e756c11849faf67c98847f1aa3b8cbe2a89b1b4: Status 404 returned error can't find the container with id 48d8dc8be93444a88b43ec968e756c11849faf67c98847f1aa3b8cbe2a89b1b4
Jul 09 23:28:39.870798 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.870715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:39.870798 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.870770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:39.871031 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:39.870847    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 23:28:39.871031 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:39.870864    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 23:28:39.871031 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:39.870926    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm podName:c4ab2689-0ddb-46e2-95b9-93710ee87dc4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:41.870906603 +0000 UTC m=+40.014837366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7b2ch" (UID: "c4ab2689-0ddb-46e2-95b9-93710ee87dc4") : configmap "kube-rbac-proxy" not found
Jul 09 23:28:39.871031 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:39.870949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:39.871031 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:39.870980    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls podName:b9160d80-c17d-491e-b077-5917e5d6be05 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:41.870965354 +0000 UTC m=+40.014896124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls") pod "dns-default-xll26" (UID: "b9160d80-c17d-491e-b077-5917e5d6be05") : secret "dns-default-metrics-tls" not found
Jul 09 23:28:39.871230 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:39.871038    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 23:28:39.871230 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:39.871088    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls podName:c4ab2689-0ddb-46e2-95b9-93710ee87dc4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:41.871075198 +0000 UTC m=+40.015005976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7b2ch" (UID: "c4ab2689-0ddb-46e2-95b9-93710ee87dc4") : secret "insights-runtime-extractor-tls" not found
Jul 09 23:28:40.612717 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:40.612682    2580 generic.go:356] "Generic (PLEG): container finished" podID="8083d0cd-a5a4-42fe-863a-e70e5b0e8a55" containerID="612ef4b119e385147094ce5aa675cc37a995c913e187549ba41d2674de4aa647" exitCode=0
Jul 09 23:28:40.613376 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:40.612771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p96p7" event={"ID":"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55","Type":"ContainerDied","Data":"612ef4b119e385147094ce5aa675cc37a995c913e187549ba41d2674de4aa647"}
Jul 09 23:28:40.614547 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:40.614522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b" event={"ID":"c48db8d5-e73a-4c05-a0a2-c2dbf7402913","Type":"ContainerStarted","Data":"48d8dc8be93444a88b43ec968e756c11849faf67c98847f1aa3b8cbe2a89b1b4"}
Jul 09 23:28:41.621382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:41.621339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p96p7" event={"ID":"8083d0cd-a5a4-42fe-863a-e70e5b0e8a55","Type":"ContainerStarted","Data":"47296a9f17c88b02321ed2231e937d583ec428dcdc8330c8d77a742a1ce99d33"}
Jul 09 23:28:41.623051 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:41.623016    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c6ztd" event={"ID":"15f14655-5e80-48b0-a2d4-81a4ddadaec6","Type":"ContainerStarted","Data":"7411e89b9afe05220e09952eb583085050a171f1a090f9e573134e8ab526752c"}
Jul 09 23:28:41.645068 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:41.645008    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p96p7" podStartSLOduration=9.579666484 podStartE2EDuration="39.644990811s" totalImagesPullingTime="30.065324327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:28:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:41.643871175 +0000 UTC m=+39.787801964" watchObservedRunningTime="2026-07-09 23:28:41.644990811 +0000 UTC m=+39.788921599"
Jul 09 23:28:41.658326 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:41.658261    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c6ztd" podStartSLOduration=1.256895673 podStartE2EDuration="3.658238941s" totalImagesPullingTime="2.401343268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:41.65764038 +0000 UTC m=+39.801571167" watchObservedRunningTime="2026-07-09 23:28:41.658238941 +0000 UTC m=+39.802169728"
Jul 09 23:28:41.889310 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:41.889223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:41.889310 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:41.889278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:41.889529 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:41.889348    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 23:28:41.889529 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:41.889403    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm podName:c4ab2689-0ddb-46e2-95b9-93710ee87dc4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:45.889388321 +0000 UTC m=+44.033319085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7b2ch" (UID: "c4ab2689-0ddb-46e2-95b9-93710ee87dc4") : configmap "kube-rbac-proxy" not found
Jul 09 23:28:41.889529 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:41.889402    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 23:28:41.889529 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:41.889470    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls podName:c4ab2689-0ddb-46e2-95b9-93710ee87dc4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:45.889451432 +0000 UTC m=+44.033382209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7b2ch" (UID: "c4ab2689-0ddb-46e2-95b9-93710ee87dc4") : secret "insights-runtime-extractor-tls" not found
Jul 09 23:28:41.889529 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:41.889498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:41.889785 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:41.889638    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 23:28:41.889785 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:41.889702    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls podName:b9160d80-c17d-491e-b077-5917e5d6be05 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:45.88969166 +0000 UTC m=+44.033622424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls") pod "dns-default-xll26" (UID: "b9160d80-c17d-491e-b077-5917e5d6be05") : secret "dns-default-metrics-tls" not found
Jul 09 23:28:42.626429 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:42.626394    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b" event={"ID":"c48db8d5-e73a-4c05-a0a2-c2dbf7402913","Type":"ContainerStarted","Data":"ecf9f631dd45910a0aa596cd56d05ba0c193588cad53a3da50cbdefd5a3df4ee"}
Jul 09 23:28:45.925562 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:45.925518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:45.926023 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:45.925608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:45.926023 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:45.925633    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 23:28:45.926023 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:45.925680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:45.926023 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:45.925717    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls podName:b9160d80-c17d-491e-b077-5917e5d6be05 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:53.925697679 +0000 UTC m=+52.069628462 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls") pod "dns-default-xll26" (UID: "b9160d80-c17d-491e-b077-5917e5d6be05") : secret "dns-default-metrics-tls" not found
Jul 09 23:28:45.926023 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:45.925741    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 23:28:45.926023 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:45.925750    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 23:28:45.926023 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:45.925779    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm podName:c4ab2689-0ddb-46e2-95b9-93710ee87dc4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:53.925768987 +0000 UTC m=+52.069699754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7b2ch" (UID: "c4ab2689-0ddb-46e2-95b9-93710ee87dc4") : configmap "kube-rbac-proxy" not found
Jul 09 23:28:45.926023 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:28:45.925812    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls podName:c4ab2689-0ddb-46e2-95b9-93710ee87dc4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:53.925795493 +0000 UTC m=+52.069726263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7b2ch" (UID: "c4ab2689-0ddb-46e2-95b9-93710ee87dc4") : secret "insights-runtime-extractor-tls" not found
Jul 09 23:28:53.991690 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:53.991640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:53.992186 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:53.991729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:53.992186 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:53.991784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:53.992314 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:53.992290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:53.994863 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:53.994839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9160d80-c17d-491e-b077-5917e5d6be05-metrics-tls\") pod \"dns-default-xll26\" (UID: \"b9160d80-c17d-491e-b077-5917e5d6be05\") " pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:53.994948 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:53.994910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4ab2689-0ddb-46e2-95b9-93710ee87dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7b2ch\" (UID: \"c4ab2689-0ddb-46e2-95b9-93710ee87dc4\") " pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:54.020801 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:54.020764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:54.040708 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:54.040647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7b2ch"
Jul 09 23:28:54.187934 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:54.187871    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-b4k4b" podStartSLOduration=13.079141351 podStartE2EDuration="15.187850781s" totalImagesPullingTime="2.10870943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:42.641127992 +0000 UTC m=+40.785058782" watchObservedRunningTime="2026-07-09 23:28:54.187850781 +0000 UTC m=+52.331781568"
Jul 09 23:28:54.188715 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:54.188693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xll26"]
Jul 09 23:28:54.191439 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:54.191413    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9160d80_c17d_491e_b077_5917e5d6be05.slice/crio-21bb9db3955c5f3caee0dca7adb466ff7741ac3564ea94561f3cea340dc57be8 WatchSource:0}: Error finding container 21bb9db3955c5f3caee0dca7adb466ff7741ac3564ea94561f3cea340dc57be8: Status 404 returned error can't find the container with id 21bb9db3955c5f3caee0dca7adb466ff7741ac3564ea94561f3cea340dc57be8
Jul 09 23:28:54.210622 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:54.210599    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7b2ch"]
Jul 09 23:28:54.213606 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:54.213583    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ab2689_0ddb_46e2_95b9_93710ee87dc4.slice/crio-cac4397f78fb3ef26d410abf5a009699bac0bbfc6b596a751672f94d99738ed0 WatchSource:0}: Error finding container cac4397f78fb3ef26d410abf5a009699bac0bbfc6b596a751672f94d99738ed0: Status 404 returned error can't find the container with id cac4397f78fb3ef26d410abf5a009699bac0bbfc6b596a751672f94d99738ed0
Jul 09 23:28:54.655204 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:54.655168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xll26" event={"ID":"b9160d80-c17d-491e-b077-5917e5d6be05","Type":"ContainerStarted","Data":"21bb9db3955c5f3caee0dca7adb466ff7741ac3564ea94561f3cea340dc57be8"}
Jul 09 23:28:54.656173 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:54.656142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7b2ch" event={"ID":"c4ab2689-0ddb-46e2-95b9-93710ee87dc4","Type":"ContainerStarted","Data":"cac4397f78fb3ef26d410abf5a009699bac0bbfc6b596a751672f94d99738ed0"}
Jul 09 23:28:58.209611 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.209567    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9"]
Jul 09 23:28:58.211524 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.211501    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-4zx77"]
Jul 09 23:28:58.211669 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.211632    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9"
Jul 09 23:28:58.214065 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.214043    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-4zx77"
Jul 09 23:28:58.214740 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.214406    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 23:28:58.214740 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.214479    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"]
Jul 09 23:28:58.214740 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.214610    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xj4j7\""
Jul 09 23:28:58.214942 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.214751    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 23:28:58.216339 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.216323    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.217584 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.217568    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 23:28:58.217688 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.217626    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 23:28:58.219705 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.219556    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 23:28:58.219705 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.219675    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 23:28:58.219863 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.219735    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-c2fs8\""
Jul 09 23:28:58.219863 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.219830    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 23:28:58.220250 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.219986    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 23:28:58.220250 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.220059    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 23:28:58.220250 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.220127    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 23:28:58.222910 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.222889    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9"]
Jul 09 23:28:58.227241 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.227193    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-4zx77"]
Jul 09 23:28:58.231265 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.231243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"]
Jul 09 23:28:58.323870 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.323833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfkmb\" (UniqueName: \"kubernetes.io/projected/971ba3dc-711d-4935-bccb-a6c465555b1c-kube-api-access-tfkmb\") pod \"downloads-68c9cbb497-4zx77\" (UID: \"971ba3dc-711d-4935-bccb-a6c465555b1c\") " pod="openshift-console/downloads-68c9cbb497-4zx77"
Jul 09 23:28:58.323870 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.323867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/437746df-bae7-40f1-baab-cacc3e62896c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.324100 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.323901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbf2s\" (UniqueName: \"kubernetes.io/projected/eb89fa74-ef33-4610-9bcc-51385a5caa89-kube-api-access-cbf2s\") pod \"managed-serviceaccount-addon-agent-77c86876cd-kb7t9\" (UID: \"eb89fa74-ef33-4610-9bcc-51385a5caa89\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9"
Jul 09 23:28:58.324100 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.323993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/437746df-bae7-40f1-baab-cacc3e62896c-ca\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.324100 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.324025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/437746df-bae7-40f1-baab-cacc3e62896c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.324100 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.324045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eb89fa74-ef33-4610-9bcc-51385a5caa89-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77c86876cd-kb7t9\" (UID: \"eb89fa74-ef33-4610-9bcc-51385a5caa89\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9"
Jul 09 23:28:58.324100 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.324085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/437746df-bae7-40f1-baab-cacc3e62896c-hub\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.324348 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.324118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/437746df-bae7-40f1-baab-cacc3e62896c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.324348 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.324187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62bd8\" (UniqueName: \"kubernetes.io/projected/437746df-bae7-40f1-baab-cacc3e62896c-kube-api-access-62bd8\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.425281 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.425242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbf2s\" (UniqueName: \"kubernetes.io/projected/eb89fa74-ef33-4610-9bcc-51385a5caa89-kube-api-access-cbf2s\") pod \"managed-serviceaccount-addon-agent-77c86876cd-kb7t9\" (UID: \"eb89fa74-ef33-4610-9bcc-51385a5caa89\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9"
Jul 09 23:28:58.425471 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.425306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/437746df-bae7-40f1-baab-cacc3e62896c-ca\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.425471 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.425330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/437746df-bae7-40f1-baab-cacc3e62896c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.425471 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.425352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eb89fa74-ef33-4610-9bcc-51385a5caa89-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77c86876cd-kb7t9\" (UID: \"eb89fa74-ef33-4610-9bcc-51385a5caa89\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9"
Jul 09 23:28:58.425471 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.425385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/437746df-bae7-40f1-baab-cacc3e62896c-hub\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.425471 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.425420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/437746df-bae7-40f1-baab-cacc3e62896c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.425750 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.425477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62bd8\" (UniqueName: \"kubernetes.io/projected/437746df-bae7-40f1-baab-cacc3e62896c-kube-api-access-62bd8\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.425750 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.425537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfkmb\" (UniqueName: \"kubernetes.io/projected/971ba3dc-711d-4935-bccb-a6c465555b1c-kube-api-access-tfkmb\") pod \"downloads-68c9cbb497-4zx77\" (UID: \"971ba3dc-711d-4935-bccb-a6c465555b1c\") " pod="openshift-console/downloads-68c9cbb497-4zx77"
Jul 09 23:28:58.425750 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.425564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/437746df-bae7-40f1-baab-cacc3e62896c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.426399 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.426337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/437746df-bae7-40f1-baab-cacc3e62896c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.428415 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.428387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/437746df-bae7-40f1-baab-cacc3e62896c-ca\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.428544 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.428525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eb89fa74-ef33-4610-9bcc-51385a5caa89-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77c86876cd-kb7t9\" (UID: \"eb89fa74-ef33-4610-9bcc-51385a5caa89\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9"
Jul 09 23:28:58.429261 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.429240    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/437746df-bae7-40f1-baab-cacc3e62896c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.429391 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.429363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/437746df-bae7-40f1-baab-cacc3e62896c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.429495 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.429436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/437746df-bae7-40f1-baab-cacc3e62896c-hub\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.434966 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.434941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfkmb\" (UniqueName: \"kubernetes.io/projected/971ba3dc-711d-4935-bccb-a6c465555b1c-kube-api-access-tfkmb\") pod \"downloads-68c9cbb497-4zx77\" (UID: \"971ba3dc-711d-4935-bccb-a6c465555b1c\") " pod="openshift-console/downloads-68c9cbb497-4zx77"
Jul 09 23:28:58.435210 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.435184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62bd8\" (UniqueName: \"kubernetes.io/projected/437746df-bae7-40f1-baab-cacc3e62896c-kube-api-access-62bd8\") pod \"cluster-proxy-proxy-agent-65547f76f9-f6wck\" (UID: \"437746df-bae7-40f1-baab-cacc3e62896c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.435777 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.435303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbf2s\" (UniqueName: \"kubernetes.io/projected/eb89fa74-ef33-4610-9bcc-51385a5caa89-kube-api-access-cbf2s\") pod \"managed-serviceaccount-addon-agent-77c86876cd-kb7t9\" (UID: \"eb89fa74-ef33-4610-9bcc-51385a5caa89\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9"
Jul 09 23:28:58.535044 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.534849    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9"
Jul 09 23:28:58.543185 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.543149    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-4zx77"
Jul 09 23:28:58.549763 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.549581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"
Jul 09 23:28:58.674809 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.674668    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xll26" event={"ID":"b9160d80-c17d-491e-b077-5917e5d6be05","Type":"ContainerStarted","Data":"eceffddaa78dd844bf711754a4d6117c241224e7fd21e4426468c4d97633cc7d"}
Jul 09 23:28:58.677003 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.676924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7b2ch" event={"ID":"c4ab2689-0ddb-46e2-95b9-93710ee87dc4","Type":"ContainerStarted","Data":"cb6bb5dd12cf16044ead308e5cc456e10c1a36aa1ed9b49d1bda011c0b8e55e9"}
Jul 09 23:28:58.709775 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.709568    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9"]
Jul 09 23:28:58.711191 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:58.711163    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb89fa74_ef33_4610_9bcc_51385a5caa89.slice/crio-e9751880c78dc9eb989a09c6eb7eb34d830fbcf8a1af9b91dcba6b3aacef76f4 WatchSource:0}: Error finding container e9751880c78dc9eb989a09c6eb7eb34d830fbcf8a1af9b91dcba6b3aacef76f4: Status 404 returned error can't find the container with id e9751880c78dc9eb989a09c6eb7eb34d830fbcf8a1af9b91dcba6b3aacef76f4
Jul 09 23:28:58.736934 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.736911    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck"]
Jul 09 23:28:58.739416 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:58.739394    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod437746df_bae7_40f1_baab_cacc3e62896c.slice/crio-661ec2c56a92984454bfadde52f4dae37b92e9a786a1a08777356d30ea46aa08 WatchSource:0}: Error finding container 661ec2c56a92984454bfadde52f4dae37b92e9a786a1a08777356d30ea46aa08: Status 404 returned error can't find the container with id 661ec2c56a92984454bfadde52f4dae37b92e9a786a1a08777356d30ea46aa08
Jul 09 23:28:58.755636 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:58.755570    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-4zx77"]
Jul 09 23:28:58.758498 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:28:58.758472    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971ba3dc_711d_4935_bccb_a6c465555b1c.slice/crio-da7d5b5fc4bf90c4ab0db996b7991742e86790473ddc20fd05333899cd4e0d34 WatchSource:0}: Error finding container da7d5b5fc4bf90c4ab0db996b7991742e86790473ddc20fd05333899cd4e0d34: Status 404 returned error can't find the container with id da7d5b5fc4bf90c4ab0db996b7991742e86790473ddc20fd05333899cd4e0d34
Jul 09 23:28:59.687780 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:59.687139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xll26" event={"ID":"b9160d80-c17d-491e-b077-5917e5d6be05","Type":"ContainerStarted","Data":"51ab15c6ab77a0d6470c48a9076ba8f91d244f1b43f15183942b0c85902fa72f"}
Jul 09 23:28:59.687780 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:59.687725    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xll26"
Jul 09 23:28:59.690592 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:59.690538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck" event={"ID":"437746df-bae7-40f1-baab-cacc3e62896c","Type":"ContainerStarted","Data":"661ec2c56a92984454bfadde52f4dae37b92e9a786a1a08777356d30ea46aa08"}
Jul 09 23:28:59.693356 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:59.693333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9" event={"ID":"eb89fa74-ef33-4610-9bcc-51385a5caa89","Type":"ContainerStarted","Data":"e9751880c78dc9eb989a09c6eb7eb34d830fbcf8a1af9b91dcba6b3aacef76f4"}
Jul 09 23:28:59.697058 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:59.697032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7b2ch" event={"ID":"c4ab2689-0ddb-46e2-95b9-93710ee87dc4","Type":"ContainerStarted","Data":"4ef0ba5456b32e2f09872263987911bd7b87a342c3674bbe60a6032b94532ac9"}
Jul 09 23:28:59.698281 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:28:59.698228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-4zx77" event={"ID":"971ba3dc-711d-4935-bccb-a6c465555b1c","Type":"ContainerStarted","Data":"da7d5b5fc4bf90c4ab0db996b7991742e86790473ddc20fd05333899cd4e0d34"}
Jul 09 23:29:03.599514 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:03.599431    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rvp85"
Jul 09 23:29:03.627044 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:03.626983    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xll26" podStartSLOduration=21.323031266 podStartE2EDuration="25.626964968s" totalImagesPullingTime="4.303933702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:59.70649611 +0000 UTC m=+57.850426918" watchObservedRunningTime="2026-07-09 23:29:03.626964968 +0000 UTC m=+61.770895755"
Jul 09 23:29:05.727096 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:05.727056    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck" event={"ID":"437746df-bae7-40f1-baab-cacc3e62896c","Type":"ContainerStarted","Data":"95553bf33e06b3c3f21fd8dfe44db638d59ed2ef385e7b788122c9af416247d2"}
Jul 09 23:29:05.727619 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:05.727115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck" event={"ID":"437746df-bae7-40f1-baab-cacc3e62896c","Type":"ContainerStarted","Data":"f5ebe403a4d7e02d98cf8eb408f4d0cf7d66fa5e593d641d1ea32e7dd2ea1190"}
Jul 09 23:29:05.727619 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:05.727137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck" event={"ID":"437746df-bae7-40f1-baab-cacc3e62896c","Type":"ContainerStarted","Data":"8a9c9aa2aff56e1b29e62b905fd8a003b048a09fa38e1465656b7f38fcbf242f"}
Jul 09 23:29:05.728645 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:05.728615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9" event={"ID":"eb89fa74-ef33-4610-9bcc-51385a5caa89","Type":"ContainerStarted","Data":"2da88bfe9cc2b0a02b049dcea5f9151e7c4e167b285432478882fbadd47cd51d"}
Jul 09 23:29:05.730919 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:05.730894    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7b2ch" event={"ID":"c4ab2689-0ddb-46e2-95b9-93710ee87dc4","Type":"ContainerStarted","Data":"12895d9a65b59d7a2caf5d10cf1873b1b3a3c71727df6d1cedcc92be0df76e89"}
Jul 09 23:29:05.743479 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:05.743420    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65547f76f9-f6wck" podStartSLOduration=1.456027876 podStartE2EDuration="7.743403146s" totalImagesPullingTime="6.28737527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:05.742564656 +0000 UTC m=+63.886495442" watchObservedRunningTime="2026-07-09 23:29:05.743403146 +0000 UTC m=+63.887333915"
Jul 09 23:29:05.759074 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:05.759020    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7b2ch" podStartSLOduration=17.086090524 podStartE2EDuration="27.759002259s" totalImagesPullingTime="10.672911735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:28:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:05.757790582 +0000 UTC m=+63.901721368" watchObservedRunningTime="2026-07-09 23:29:05.759002259 +0000 UTC m=+63.902933029"
Jul 09 23:29:05.770700 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:05.770642    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c86876cd-kb7t9" podStartSLOduration=1.48480182 podStartE2EDuration="7.770627348s" totalImagesPullingTime="6.285825528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:05.770581738 +0000 UTC m=+63.914512523" watchObservedRunningTime="2026-07-09 23:29:05.770627348 +0000 UTC m=+63.914558132"
Jul 09 23:29:08.127281 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.127230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs\") pod \"network-metrics-daemon-m5bj7\" (UID: \"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8\") " pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:29:08.129632 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.129604    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 23:29:08.140086 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.140053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7add64-a9cc-4acc-9102-a8c5efbbc5a8-metrics-certs\") pod \"network-metrics-daemon-m5bj7\" (UID: \"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8\") " pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:29:08.227640 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.227603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:29:08.227835 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.227664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdwx\" (UniqueName: \"kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx\") pod \"network-check-target-hgsp8\" (UID: \"d10a13cc-9e45-49e7-a53b-e03df17e48e7\") " pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:29:08.230034 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.230005    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 23:29:08.230196 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.230176    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 23:29:08.240231 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.240206    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 23:29:08.241235 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.241213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe6a62a4-df2a-4d08-b134-213d033b0e28-original-pull-secret\") pod \"global-pull-secret-syncer-tmmgg\" (UID: \"fe6a62a4-df2a-4d08-b134-213d033b0e28\") " pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:29:08.251851 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.251823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdwx\" (UniqueName: \"kubernetes.io/projected/d10a13cc-9e45-49e7-a53b-e03df17e48e7-kube-api-access-nxdwx\") pod \"network-check-target-hgsp8\" (UID: \"d10a13cc-9e45-49e7-a53b-e03df17e48e7\") " pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:29:08.426254 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.426180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmmgg"
Jul 09 23:29:08.435806 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.435764    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8rpsb\""
Jul 09 23:29:08.441826 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.441806    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r9l9m\""
Jul 09 23:29:08.443681 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.443623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:29:08.450928 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.450904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5bj7"
Jul 09 23:29:08.589794 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.588103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tmmgg"]
Jul 09 23:29:08.592012 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:29:08.591979    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6a62a4_df2a_4d08_b134_213d033b0e28.slice/crio-9741c79ae554255365b469387876bc7a8e06c2d1c81eb1efd47c55e025a845d5 WatchSource:0}: Error finding container 9741c79ae554255365b469387876bc7a8e06c2d1c81eb1efd47c55e025a845d5: Status 404 returned error can't find the container with id 9741c79ae554255365b469387876bc7a8e06c2d1c81eb1efd47c55e025a845d5
Jul 09 23:29:08.607454 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.607412    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hgsp8"]
Jul 09 23:29:08.610784 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:29:08.610747    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10a13cc_9e45_49e7_a53b_e03df17e48e7.slice/crio-8c8ded229ace30caa24255dde03534016e4dfd7012b332a86d59f113a683734a WatchSource:0}: Error finding container 8c8ded229ace30caa24255dde03534016e4dfd7012b332a86d59f113a683734a: Status 404 returned error can't find the container with id 8c8ded229ace30caa24255dde03534016e4dfd7012b332a86d59f113a683734a
Jul 09 23:29:08.629784 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.629754    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m5bj7"]
Jul 09 23:29:08.633359 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:29:08.633332    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7add64_a9cc_4acc_9102_a8c5efbbc5a8.slice/crio-696bc36988cb89ed12ab33e3705584b1a01b8cd8791aa1b6280ea5daad4a8f1c WatchSource:0}: Error finding container 696bc36988cb89ed12ab33e3705584b1a01b8cd8791aa1b6280ea5daad4a8f1c: Status 404 returned error can't find the container with id 696bc36988cb89ed12ab33e3705584b1a01b8cd8791aa1b6280ea5daad4a8f1c
Jul 09 23:29:08.742641 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.742546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hgsp8" event={"ID":"d10a13cc-9e45-49e7-a53b-e03df17e48e7","Type":"ContainerStarted","Data":"8c8ded229ace30caa24255dde03534016e4dfd7012b332a86d59f113a683734a"}
Jul 09 23:29:08.743937 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.743905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tmmgg" event={"ID":"fe6a62a4-df2a-4d08-b134-213d033b0e28","Type":"ContainerStarted","Data":"9741c79ae554255365b469387876bc7a8e06c2d1c81eb1efd47c55e025a845d5"}
Jul 09 23:29:08.745208 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:08.745183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m5bj7" event={"ID":"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8","Type":"ContainerStarted","Data":"696bc36988cb89ed12ab33e3705584b1a01b8cd8791aa1b6280ea5daad4a8f1c"}
Jul 09 23:29:09.708914 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:09.708289    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xll26"
Jul 09 23:29:16.334235 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.334200    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rq6t6"]
Jul 09 23:29:16.502894 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.502860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.506204 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.506177    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 23:29:16.506356 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.506219    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 23:29:16.506356 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.506178    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 23:29:16.506465 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.506416    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 23:29:16.506465 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.506461    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 23:29:16.506597 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.506578    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 23:29:16.508499 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.508016    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mhc8b\""
Jul 09 23:29:16.586249 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.586159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.586249 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.586215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-tls\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.586436 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.586257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-wtmp\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.586436 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.586283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-textfile\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.586436 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.586307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.586436 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.586339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbec990d-76e2-4c8e-8955-24270452b2d3-metrics-client-ca\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.586436 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.586365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cbec990d-76e2-4c8e-8955-24270452b2d3-sys\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.586436 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.586393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cbec990d-76e2-4c8e-8955-24270452b2d3-root\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.586436 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.586415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6djn9\" (UniqueName: \"kubernetes.io/projected/cbec990d-76e2-4c8e-8955-24270452b2d3-kube-api-access-6djn9\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.687172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.687095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.687349 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.687179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-tls\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.687349 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.687223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-wtmp\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.687349 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.687248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-textfile\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.687349 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.687272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.687349 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.687305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbec990d-76e2-4c8e-8955-24270452b2d3-metrics-client-ca\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.687349 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.687335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cbec990d-76e2-4c8e-8955-24270452b2d3-sys\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.688839 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.687367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cbec990d-76e2-4c8e-8955-24270452b2d3-root\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.688839 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.687391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6djn9\" (UniqueName: \"kubernetes.io/projected/cbec990d-76e2-4c8e-8955-24270452b2d3-kube-api-access-6djn9\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.688839 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:29:16.687771    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 09 23:29:16.688839 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:29:16.687844    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-tls podName:cbec990d-76e2-4c8e-8955-24270452b2d3 nodeName:}" failed. No retries permitted until 2026-07-09 23:29:17.187822216 +0000 UTC m=+75.331752993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-tls") pod "node-exporter-rq6t6" (UID: "cbec990d-76e2-4c8e-8955-24270452b2d3") : secret "node-exporter-tls" not found
Jul 09 23:29:16.688839 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.687852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-textfile\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.688839 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.687997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-wtmp\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.688839 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.688087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cbec990d-76e2-4c8e-8955-24270452b2d3-sys\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.688839 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.688488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.688839 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.688546    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cbec990d-76e2-4c8e-8955-24270452b2d3-root\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.688839 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.688630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbec990d-76e2-4c8e-8955-24270452b2d3-metrics-client-ca\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.691137 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.691078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:16.698378 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:16.698282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6djn9\" (UniqueName: \"kubernetes.io/projected/cbec990d-76e2-4c8e-8955-24270452b2d3-kube-api-access-6djn9\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:17.191614 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:17.191573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-tls\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:17.191825 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:29:17.191733    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 09 23:29:17.191825 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:29:17.191802    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-tls podName:cbec990d-76e2-4c8e-8955-24270452b2d3 nodeName:}" failed. No retries permitted until 2026-07-09 23:29:18.191781901 +0000 UTC m=+76.335712670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-tls") pod "node-exporter-rq6t6" (UID: "cbec990d-76e2-4c8e-8955-24270452b2d3") : secret "node-exporter-tls" not found
Jul 09 23:29:18.201262 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.201225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-tls\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:18.203983 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.203950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cbec990d-76e2-4c8e-8955-24270452b2d3-node-exporter-tls\") pod \"node-exporter-rq6t6\" (UID: \"cbec990d-76e2-4c8e-8955-24270452b2d3\") " pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:18.316464 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.316422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rq6t6"
Jul 09 23:29:18.327611 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.327566    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7487767f58-9jstb"]
Jul 09 23:29:18.376754 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.376708    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7487767f58-9jstb"]
Jul 09 23:29:18.376883 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.376816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.379467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.379443    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 09 23:29:18.379628 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.379444    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 09 23:29:18.379763 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.379707    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dsga8btoe96u0\""
Jul 09 23:29:18.379847 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.379822    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 09 23:29:18.379929 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.379879    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 09 23:29:18.379929 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.379888    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 09 23:29:18.380026 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.379934    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m7z2x\""
Jul 09 23:29:18.503266 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.503185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnvxk\" (UniqueName: \"kubernetes.io/projected/e319659e-5d27-4ec3-8a9e-05b77d06b003-kube-api-access-cnvxk\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.503266 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.503230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-grpc-tls\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.503266 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.503259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.503529 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.503352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.503529 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.503410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e319659e-5d27-4ec3-8a9e-05b77d06b003-metrics-client-ca\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.503529 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.503444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-tls\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.503529 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.503486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.503529 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.503508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.604306 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.604266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e319659e-5d27-4ec3-8a9e-05b77d06b003-metrics-client-ca\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.604306 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.604311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-tls\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.604536 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.604358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.604536 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.604393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.604536 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.604420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnvxk\" (UniqueName: \"kubernetes.io/projected/e319659e-5d27-4ec3-8a9e-05b77d06b003-kube-api-access-cnvxk\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.604536 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.604450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-grpc-tls\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.604536 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.604486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.604782 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.604646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.605109 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.605069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e319659e-5d27-4ec3-8a9e-05b77d06b003-metrics-client-ca\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.607134 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.607098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.607270 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.607147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-tls\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.607436 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.607401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.607436 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.607434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-grpc-tls\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.607600 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.607486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.607600 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.607493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e319659e-5d27-4ec3-8a9e-05b77d06b003-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.612093 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.612071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnvxk\" (UniqueName: \"kubernetes.io/projected/e319659e-5d27-4ec3-8a9e-05b77d06b003-kube-api-access-cnvxk\") pod \"thanos-querier-7487767f58-9jstb\" (UID: \"e319659e-5d27-4ec3-8a9e-05b77d06b003\") " pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:18.692278 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:18.692241    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:20.778371 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.778133    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7487767f58-9jstb"]
Jul 09 23:29:20.780269 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:29:20.780236    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode319659e_5d27_4ec3_8a9e_05b77d06b003.slice/crio-66b8d0551a452a107d689b7d8d469f126405ad430f7cae9e175a4e8749b40201 WatchSource:0}: Error finding container 66b8d0551a452a107d689b7d8d469f126405ad430f7cae9e175a4e8749b40201: Status 404 returned error can't find the container with id 66b8d0551a452a107d689b7d8d469f126405ad430f7cae9e175a4e8749b40201
Jul 09 23:29:20.792870 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.792841    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rq6t6" event={"ID":"cbec990d-76e2-4c8e-8955-24270452b2d3","Type":"ContainerStarted","Data":"6a9c0d5f00654fe15024d5e6060cbd0c57b02167589faa8bff6254aad38913ef"}
Jul 09 23:29:20.794425 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.794404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-4zx77" event={"ID":"971ba3dc-711d-4935-bccb-a6c465555b1c","Type":"ContainerStarted","Data":"28ca52f06136d9fef9275ab586c490081083081a40e376fa9b929c5b846cd357"}
Jul 09 23:29:20.794830 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.794803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-4zx77"
Jul 09 23:29:20.797578 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.796421    2580 patch_prober.go:28] interesting pod/downloads-68c9cbb497-4zx77 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.11:8080/\": dial tcp 10.134.0.11:8080: connect: connection refused" start-of-body=
Jul 09 23:29:20.797578 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.796469    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-68c9cbb497-4zx77" podUID="971ba3dc-711d-4935-bccb-a6c465555b1c" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.11:8080/\": dial tcp 10.134.0.11:8080: connect: connection refused"
Jul 09 23:29:20.797578 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.796563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m5bj7" event={"ID":"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8","Type":"ContainerStarted","Data":"ccefc37f8f0d9c516868805206879f277cd03fb16ef8f41eadc7aabba1d8cdbe"}
Jul 09 23:29:20.800035 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.800009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hgsp8" event={"ID":"d10a13cc-9e45-49e7-a53b-e03df17e48e7","Type":"ContainerStarted","Data":"44e2858ffbdb4a6d9160d03033097b8d0d782b42779d7f6bd7473a6fb5d2314b"}
Jul 09 23:29:20.800451 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.800412    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:29:20.801644 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.801613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tmmgg" event={"ID":"fe6a62a4-df2a-4d08-b134-213d033b0e28","Type":"ContainerStarted","Data":"7637fe773b7c90624ceeb1b5e2df66214455dda9fdbe9c140078fe7b40c1720b"}
Jul 09 23:29:20.802788 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.802756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7487767f58-9jstb" event={"ID":"e319659e-5d27-4ec3-8a9e-05b77d06b003","Type":"ContainerStarted","Data":"66b8d0551a452a107d689b7d8d469f126405ad430f7cae9e175a4e8749b40201"}
Jul 09 23:29:20.818877 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.818803    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-4zx77" podStartSLOduration=0.979602424 podStartE2EDuration="22.818782637s" totalImagesPullingTime="21.839180213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:20.8179888 +0000 UTC m=+78.961919585" watchObservedRunningTime="2026-07-09 23:29:20.818782637 +0000 UTC m=+78.962713421"
Jul 09 23:29:20.832898 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.832846    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tmmgg" podStartSLOduration=65.820881048 podStartE2EDuration="1m17.832817569s" totalImagesPullingTime="12.011936521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:20.831245074 +0000 UTC m=+78.975175876" watchObservedRunningTime="2026-07-09 23:29:20.832817569 +0000 UTC m=+78.976748353"
Jul 09 23:29:20.852544 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:20.852491    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hgsp8" podStartSLOduration=65.864741704 podStartE2EDuration="1m17.85247415s" totalImagesPullingTime="11.987732446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:20.844495511 +0000 UTC m=+78.988426296" watchObservedRunningTime="2026-07-09 23:29:20.85247415 +0000 UTC m=+78.996404934"
Jul 09 23:29:21.149428 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.149388    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d"]
Jul 09 23:29:21.152485 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.152156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d"
Jul 09 23:29:21.155020 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.154993    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 09 23:29:21.155145 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.155079    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-74qpn\""
Jul 09 23:29:21.163928 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.163895    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d"]
Jul 09 23:29:21.331024 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.330854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e84df855-0915-471f-8879-b7e0614a0d59-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-tr59d\" (UID: \"e84df855-0915-471f-8879-b7e0614a0d59\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d"
Jul 09 23:29:21.331024 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.330912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58qg\" (UniqueName: \"kubernetes.io/projected/e84df855-0915-471f-8879-b7e0614a0d59-kube-api-access-d58qg\") pod \"monitoring-plugin-5f68df7f58-tr59d\" (UID: \"e84df855-0915-471f-8879-b7e0614a0d59\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d"
Jul 09 23:29:21.432042 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.431416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e84df855-0915-471f-8879-b7e0614a0d59-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-tr59d\" (UID: \"e84df855-0915-471f-8879-b7e0614a0d59\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d"
Jul 09 23:29:21.432042 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.431480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58qg\" (UniqueName: \"kubernetes.io/projected/e84df855-0915-471f-8879-b7e0614a0d59-kube-api-access-d58qg\") pod \"monitoring-plugin-5f68df7f58-tr59d\" (UID: \"e84df855-0915-471f-8879-b7e0614a0d59\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d"
Jul 09 23:29:21.434911 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.434870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e84df855-0915-471f-8879-b7e0614a0d59-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-tr59d\" (UID: \"e84df855-0915-471f-8879-b7e0614a0d59\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d"
Jul 09 23:29:21.439244 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.439188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58qg\" (UniqueName: \"kubernetes.io/projected/e84df855-0915-471f-8879-b7e0614a0d59-kube-api-access-d58qg\") pod \"monitoring-plugin-5f68df7f58-tr59d\" (UID: \"e84df855-0915-471f-8879-b7e0614a0d59\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d"
Jul 09 23:29:21.466374 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.466159    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d"
Jul 09 23:29:21.812476 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.812440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m5bj7" event={"ID":"1e7add64-a9cc-4acc-9102-a8c5efbbc5a8","Type":"ContainerStarted","Data":"0fb455b47e69d08a21459e009b92ea07cc95143ea7839add4d73b2b4a22e3ac7"}
Jul 09 23:29:21.827528 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.827356    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-4zx77"
Jul 09 23:29:21.827714 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:21.827549    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m5bj7" podStartSLOduration=67.86384291 podStartE2EDuration="1m19.827534569s" totalImagesPullingTime="11.963691659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:21.826242071 +0000 UTC m=+79.970172857" watchObservedRunningTime="2026-07-09 23:29:21.827534569 +0000 UTC m=+79.971465356"
Jul 09 23:29:22.022183 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:22.021999    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d"]
Jul 09 23:29:22.027615 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:29:22.027573    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84df855_0915_471f_8879_b7e0614a0d59.slice/crio-8f405cd04b442c00ab160df19b70612eb25a49fac69dd61d4050b264b0198975 WatchSource:0}: Error finding container 8f405cd04b442c00ab160df19b70612eb25a49fac69dd61d4050b264b0198975: Status 404 returned error can't find the container with id 8f405cd04b442c00ab160df19b70612eb25a49fac69dd61d4050b264b0198975
Jul 09 23:29:22.819330 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:22.819237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rq6t6" event={"ID":"cbec990d-76e2-4c8e-8955-24270452b2d3","Type":"ContainerStarted","Data":"6de648cedeea3fcbaf502eed8c62676e599ef395a3b68fcaccf742a92efe4d2e"}
Jul 09 23:29:22.820914 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:22.820875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d" event={"ID":"e84df855-0915-471f-8879-b7e0614a0d59","Type":"ContainerStarted","Data":"8f405cd04b442c00ab160df19b70612eb25a49fac69dd61d4050b264b0198975"}
Jul 09 23:29:23.828414 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:23.828351    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7487767f58-9jstb" event={"ID":"e319659e-5d27-4ec3-8a9e-05b77d06b003","Type":"ContainerStarted","Data":"442af4332345000e34d7c8a860319eda03fdc7d4b4a32850e5d03125c1e1f6bf"}
Jul 09 23:29:23.828920 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:23.828499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7487767f58-9jstb" event={"ID":"e319659e-5d27-4ec3-8a9e-05b77d06b003","Type":"ContainerStarted","Data":"dcabef80cdce56197cb2e0122fc905fb948d193106c11c1d9c6fd022cc73028a"}
Jul 09 23:29:23.828920 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:23.828549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7487767f58-9jstb" event={"ID":"e319659e-5d27-4ec3-8a9e-05b77d06b003","Type":"ContainerStarted","Data":"17ae2b6c6a94ad2d9d858a4750e6b1bdd8a408512333fc55fc3c1568a3f9879d"}
Jul 09 23:29:23.830459 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:23.830270    2580 generic.go:356] "Generic (PLEG): container finished" podID="cbec990d-76e2-4c8e-8955-24270452b2d3" containerID="6de648cedeea3fcbaf502eed8c62676e599ef395a3b68fcaccf742a92efe4d2e" exitCode=0
Jul 09 23:29:23.830459 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:23.830362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rq6t6" event={"ID":"cbec990d-76e2-4c8e-8955-24270452b2d3","Type":"ContainerDied","Data":"6de648cedeea3fcbaf502eed8c62676e599ef395a3b68fcaccf742a92efe4d2e"}
Jul 09 23:29:24.837795 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:24.837757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rq6t6" event={"ID":"cbec990d-76e2-4c8e-8955-24270452b2d3","Type":"ContainerStarted","Data":"7db470096a6e6937d5ff195ac34772b6ade7564e5cd9cb6e816a510e36a05f51"}
Jul 09 23:29:24.838236 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:24.837823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rq6t6" event={"ID":"cbec990d-76e2-4c8e-8955-24270452b2d3","Type":"ContainerStarted","Data":"61c3f648764c7181d0569f3c46ff387d2919d19dd205ca16a1ff24f40911d677"}
Jul 09 23:29:24.855947 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:24.855890    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rq6t6" podStartSLOduration=7.142572288 podStartE2EDuration="8.855870811s" totalImagesPullingTime="1.621054843s" totalInitContainerRuntime="92.24368ms" isStatefulPod=true podCreationTimestamp="2026-07-09 23:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:24.854785856 +0000 UTC m=+82.998716641" watchObservedRunningTime="2026-07-09 23:29:24.855870811 +0000 UTC m=+82.999801599"
Jul 09 23:29:25.845021 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:25.844948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d" event={"ID":"e84df855-0915-471f-8879-b7e0614a0d59","Type":"ContainerStarted","Data":"b1c8b93c5caa0a9cf0bf496e677cd6206a932c4357cc28c035ae6d18ad6e53ff"}
Jul 09 23:29:25.845791 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:25.845769    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d"
Jul 09 23:29:25.848469 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:25.848375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7487767f58-9jstb" event={"ID":"e319659e-5d27-4ec3-8a9e-05b77d06b003","Type":"ContainerStarted","Data":"eeef75c98d4f1a05e3fc58df7c294308c3464a8de42036f60eaeb978a169515d"}
Jul 09 23:29:25.850760 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:25.850739    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d"
Jul 09 23:29:25.860731 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:25.860684    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tr59d" podStartSLOduration=2.159180842 podStartE2EDuration="4.860646113s" totalImagesPullingTime="2.701465271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:25.858563837 +0000 UTC m=+84.002494623" watchObservedRunningTime="2026-07-09 23:29:25.860646113 +0000 UTC m=+84.004576898"
Jul 09 23:29:26.856584 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:26.856538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7487767f58-9jstb" event={"ID":"e319659e-5d27-4ec3-8a9e-05b77d06b003","Type":"ContainerStarted","Data":"373ab4d18da6a49198f021e8dea7233cbce999348c2f10bee623a87a34fbbcb6"}
Jul 09 23:29:26.857062 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:26.856625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7487767f58-9jstb" event={"ID":"e319659e-5d27-4ec3-8a9e-05b77d06b003","Type":"ContainerStarted","Data":"1e90bfac30749a83175d2e0d92177071c47f2823536da8e957acf2c462e70562"}
Jul 09 23:29:26.878403 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:26.878349    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7487767f58-9jstb" podStartSLOduration=4.35420718 podStartE2EDuration="8.87833258s" totalImagesPullingTime="4.5241254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:29:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:26.876628331 +0000 UTC m=+85.020559115" watchObservedRunningTime="2026-07-09 23:29:26.87833258 +0000 UTC m=+85.022263364"
Jul 09 23:29:27.861169 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:27.861130    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:28.872091 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:28.872062    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7487767f58-9jstb"
Jul 09 23:29:51.815696 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:29:51.815646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hgsp8"
Jul 09 23:31:55.095496 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.095460    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"]
Jul 09 23:31:55.097467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.097451    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"
Jul 09 23:31:55.099780 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.099760    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dpkgt\""
Jul 09 23:31:55.099780 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.099775    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 23:31:55.100761 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.100741    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 23:31:55.106021 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.105998    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"]
Jul 09 23:31:55.150474 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.150441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp9g2\" (UniqueName: \"kubernetes.io/projected/19a5a275-9e1e-4694-aa50-89b96138e2b3-kube-api-access-gp9g2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq\" (UID: \"19a5a275-9e1e-4694-aa50-89b96138e2b3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"
Jul 09 23:31:55.150474 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.150490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19a5a275-9e1e-4694-aa50-89b96138e2b3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq\" (UID: \"19a5a275-9e1e-4694-aa50-89b96138e2b3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"
Jul 09 23:31:55.150770 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.150628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a5a275-9e1e-4694-aa50-89b96138e2b3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq\" (UID: \"19a5a275-9e1e-4694-aa50-89b96138e2b3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"
Jul 09 23:31:55.251770 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.251735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a5a275-9e1e-4694-aa50-89b96138e2b3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq\" (UID: \"19a5a275-9e1e-4694-aa50-89b96138e2b3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"
Jul 09 23:31:55.251923 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.251781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp9g2\" (UniqueName: \"kubernetes.io/projected/19a5a275-9e1e-4694-aa50-89b96138e2b3-kube-api-access-gp9g2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq\" (UID: \"19a5a275-9e1e-4694-aa50-89b96138e2b3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"
Jul 09 23:31:55.251923 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.251913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19a5a275-9e1e-4694-aa50-89b96138e2b3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq\" (UID: \"19a5a275-9e1e-4694-aa50-89b96138e2b3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"
Jul 09 23:31:55.252179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.252156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a5a275-9e1e-4694-aa50-89b96138e2b3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq\" (UID: \"19a5a275-9e1e-4694-aa50-89b96138e2b3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"
Jul 09 23:31:55.252253 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.252211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19a5a275-9e1e-4694-aa50-89b96138e2b3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq\" (UID: \"19a5a275-9e1e-4694-aa50-89b96138e2b3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"
Jul 09 23:31:55.264808 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.264776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp9g2\" (UniqueName: \"kubernetes.io/projected/19a5a275-9e1e-4694-aa50-89b96138e2b3-kube-api-access-gp9g2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq\" (UID: \"19a5a275-9e1e-4694-aa50-89b96138e2b3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"
Jul 09 23:31:55.407648 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.407557    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"
Jul 09 23:31:55.538351 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:55.538315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"]
Jul 09 23:31:55.541299 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:31:55.541264    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a5a275_9e1e_4694_aa50_89b96138e2b3.slice/crio-6a1891563a2f69dad5a95225907305d297b14fe4765ac9b5587507c9a007286d WatchSource:0}: Error finding container 6a1891563a2f69dad5a95225907305d297b14fe4765ac9b5587507c9a007286d: Status 404 returned error can't find the container with id 6a1891563a2f69dad5a95225907305d297b14fe4765ac9b5587507c9a007286d
Jul 09 23:31:56.365492 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:31:56.365456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq" event={"ID":"19a5a275-9e1e-4694-aa50-89b96138e2b3","Type":"ContainerStarted","Data":"6a1891563a2f69dad5a95225907305d297b14fe4765ac9b5587507c9a007286d"}
Jul 09 23:32:02.386604 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:02.386563    2580 generic.go:356] "Generic (PLEG): container finished" podID="19a5a275-9e1e-4694-aa50-89b96138e2b3" containerID="b3ce27dccc4bdb8aa1c85398462cdfecaa9b1508e67498334aa1e9c1c4f58787" exitCode=0
Jul 09 23:32:02.387242 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:02.386647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq" event={"ID":"19a5a275-9e1e-4694-aa50-89b96138e2b3","Type":"ContainerDied","Data":"b3ce27dccc4bdb8aa1c85398462cdfecaa9b1508e67498334aa1e9c1c4f58787"}
Jul 09 23:32:03.734981 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:03.734948    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5q4b7"]
Jul 09 23:32:03.740424 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:03.740389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5q4b7"
Jul 09 23:32:03.744463 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:03.744427    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-mcsjh\""
Jul 09 23:32:03.744623 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:03.744479    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 09 23:32:03.745509 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:03.745491    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 09 23:32:03.751350 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:03.751324    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5q4b7"]
Jul 09 23:32:03.825170 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:03.825123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0da27edf-b91b-4708-9f1f-cbe207444744-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5q4b7\" (UID: \"0da27edf-b91b-4708-9f1f-cbe207444744\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5q4b7"
Jul 09 23:32:03.825343 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:03.825190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttp5q\" (UniqueName: \"kubernetes.io/projected/0da27edf-b91b-4708-9f1f-cbe207444744-kube-api-access-ttp5q\") pod \"cert-manager-webhook-6486ffd997-5q4b7\" (UID: \"0da27edf-b91b-4708-9f1f-cbe207444744\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5q4b7"
Jul 09 23:32:03.925513 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:03.925481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0da27edf-b91b-4708-9f1f-cbe207444744-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5q4b7\" (UID: \"0da27edf-b91b-4708-9f1f-cbe207444744\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5q4b7"
Jul 09 23:32:03.925709 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:03.925533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttp5q\" (UniqueName: \"kubernetes.io/projected/0da27edf-b91b-4708-9f1f-cbe207444744-kube-api-access-ttp5q\") pod \"cert-manager-webhook-6486ffd997-5q4b7\" (UID: \"0da27edf-b91b-4708-9f1f-cbe207444744\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5q4b7"
Jul 09 23:32:03.933052 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:03.933026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0da27edf-b91b-4708-9f1f-cbe207444744-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5q4b7\" (UID: \"0da27edf-b91b-4708-9f1f-cbe207444744\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5q4b7"
Jul 09 23:32:03.933333 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:03.933309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttp5q\" (UniqueName: \"kubernetes.io/projected/0da27edf-b91b-4708-9f1f-cbe207444744-kube-api-access-ttp5q\") pod \"cert-manager-webhook-6486ffd997-5q4b7\" (UID: \"0da27edf-b91b-4708-9f1f-cbe207444744\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5q4b7"
Jul 09 23:32:04.050798 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:04.050763    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5q4b7"
Jul 09 23:32:04.402447 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:04.402408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq" event={"ID":"19a5a275-9e1e-4694-aa50-89b96138e2b3","Type":"ContainerStarted","Data":"0408def5b868b09334804e74fb14aa601f8d72cd76652438da18c17d9b0e8df8"}
Jul 09 23:32:04.417350 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:32:04.417321    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0da27edf_b91b_4708_9f1f_cbe207444744.slice/crio-a49ee75e48b3dc381188a468c7de1e2613999151d7169ba9c79b0678a9893829 WatchSource:0}: Error finding container a49ee75e48b3dc381188a468c7de1e2613999151d7169ba9c79b0678a9893829: Status 404 returned error can't find the container with id a49ee75e48b3dc381188a468c7de1e2613999151d7169ba9c79b0678a9893829
Jul 09 23:32:04.418248 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:04.418182    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5q4b7"]
Jul 09 23:32:05.409582 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:05.409363    2580 generic.go:356] "Generic (PLEG): container finished" podID="19a5a275-9e1e-4694-aa50-89b96138e2b3" containerID="0408def5b868b09334804e74fb14aa601f8d72cd76652438da18c17d9b0e8df8" exitCode=0
Jul 09 23:32:05.409582 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:05.409517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq" event={"ID":"19a5a275-9e1e-4694-aa50-89b96138e2b3","Type":"ContainerDied","Data":"0408def5b868b09334804e74fb14aa601f8d72cd76652438da18c17d9b0e8df8"}
Jul 09 23:32:05.411637 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:05.411607    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5q4b7" event={"ID":"0da27edf-b91b-4708-9f1f-cbe207444744","Type":"ContainerStarted","Data":"a49ee75e48b3dc381188a468c7de1e2613999151d7169ba9c79b0678a9893829"}
Jul 09 23:32:08.424385 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:08.424332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5q4b7" event={"ID":"0da27edf-b91b-4708-9f1f-cbe207444744","Type":"ContainerStarted","Data":"1abed073262b2ca066ed1d588132880540339bf4bd6c20dae548f0dee37c255b"}
Jul 09 23:32:08.424919 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:08.424493    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5q4b7"
Jul 09 23:32:08.439940 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:08.439882    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5q4b7" podStartSLOduration=1.941710401 podStartE2EDuration="5.43986178s" totalImagesPullingTime="3.498151379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:32:08.439348832 +0000 UTC m=+246.583279650" watchObservedRunningTime="2026-07-09 23:32:08.43986178 +0000 UTC m=+246.583792583"
Jul 09 23:32:14.431162 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:14.431132    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5q4b7"
Jul 09 23:32:15.452242 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:15.452211    2580 generic.go:356] "Generic (PLEG): container finished" podID="19a5a275-9e1e-4694-aa50-89b96138e2b3" containerID="3de95f43026513709dbc2d27e9afbb76d5c233c2e90cef7f12b64f09b82e4437" exitCode=0
Jul 09 23:32:15.452622 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:15.452305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq" event={"ID":"19a5a275-9e1e-4694-aa50-89b96138e2b3","Type":"ContainerDied","Data":"3de95f43026513709dbc2d27e9afbb76d5c233c2e90cef7f12b64f09b82e4437"}
Jul 09 23:32:16.577016 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:16.576991    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"
Jul 09 23:32:16.730115 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:16.729217    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19a5a275-9e1e-4694-aa50-89b96138e2b3-util\") pod \"19a5a275-9e1e-4694-aa50-89b96138e2b3\" (UID: \"19a5a275-9e1e-4694-aa50-89b96138e2b3\") "
Jul 09 23:32:16.730115 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:16.729279    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a5a275-9e1e-4694-aa50-89b96138e2b3-bundle\") pod \"19a5a275-9e1e-4694-aa50-89b96138e2b3\" (UID: \"19a5a275-9e1e-4694-aa50-89b96138e2b3\") "
Jul 09 23:32:16.730115 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:16.729327    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp9g2\" (UniqueName: \"kubernetes.io/projected/19a5a275-9e1e-4694-aa50-89b96138e2b3-kube-api-access-gp9g2\") pod \"19a5a275-9e1e-4694-aa50-89b96138e2b3\" (UID: \"19a5a275-9e1e-4694-aa50-89b96138e2b3\") "
Jul 09 23:32:16.733392 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:16.732908    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a5a275-9e1e-4694-aa50-89b96138e2b3-kube-api-access-gp9g2" (OuterVolumeSpecName: "kube-api-access-gp9g2") pod "19a5a275-9e1e-4694-aa50-89b96138e2b3" (UID: "19a5a275-9e1e-4694-aa50-89b96138e2b3"). InnerVolumeSpecName "kube-api-access-gp9g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:32:16.733392 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:16.733175    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a5a275-9e1e-4694-aa50-89b96138e2b3-bundle" (OuterVolumeSpecName: "bundle") pod "19a5a275-9e1e-4694-aa50-89b96138e2b3" (UID: "19a5a275-9e1e-4694-aa50-89b96138e2b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:32:16.739578 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:16.739383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a5a275-9e1e-4694-aa50-89b96138e2b3-util" (OuterVolumeSpecName: "util") pod "19a5a275-9e1e-4694-aa50-89b96138e2b3" (UID: "19a5a275-9e1e-4694-aa50-89b96138e2b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:32:16.830855 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:16.830819    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19a5a275-9e1e-4694-aa50-89b96138e2b3-util\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:32:16.830855 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:16.830853    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a5a275-9e1e-4694-aa50-89b96138e2b3-bundle\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:32:16.831044 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:16.830867    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp9g2\" (UniqueName: \"kubernetes.io/projected/19a5a275-9e1e-4694-aa50-89b96138e2b3-kube-api-access-gp9g2\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:32:17.461352 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:17.461325    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq"
Jul 09 23:32:17.461565 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:17.461324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fp9fkq" event={"ID":"19a5a275-9e1e-4694-aa50-89b96138e2b3","Type":"ContainerDied","Data":"6a1891563a2f69dad5a95225907305d297b14fe4765ac9b5587507c9a007286d"}
Jul 09 23:32:17.461565 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:17.461445    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a1891563a2f69dad5a95225907305d297b14fe4765ac9b5587507c9a007286d"
Jul 09 23:32:25.752056 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.751973    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq"]
Jul 09 23:32:25.752407 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.752227    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19a5a275-9e1e-4694-aa50-89b96138e2b3" containerName="extract"
Jul 09 23:32:25.752407 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.752241    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19a5a275-9e1e-4694-aa50-89b96138e2b3" containerName="extract"
Jul 09 23:32:25.752407 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.752259    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19a5a275-9e1e-4694-aa50-89b96138e2b3" containerName="pull"
Jul 09 23:32:25.752407 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.752266    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19a5a275-9e1e-4694-aa50-89b96138e2b3" containerName="pull"
Jul 09 23:32:25.752407 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.752279    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19a5a275-9e1e-4694-aa50-89b96138e2b3" containerName="util"
Jul 09 23:32:25.752407 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.752284    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19a5a275-9e1e-4694-aa50-89b96138e2b3" containerName="util"
Jul 09 23:32:25.752407 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.752336    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19a5a275-9e1e-4694-aa50-89b96138e2b3" containerName="extract"
Jul 09 23:32:25.754218 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.754201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq"
Jul 09 23:32:25.756883 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.756863    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 09 23:32:25.756883 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.756880    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-qmd85\""
Jul 09 23:32:25.757048 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.756920    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 09 23:32:25.761963 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.761941    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq"]
Jul 09 23:32:25.896718 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.896681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdk5z\" (UniqueName: \"kubernetes.io/projected/a7d8f739-da68-4dcc-a17e-47ca4a85f8fe-kube-api-access-rdk5z\") pod \"openshift-lws-operator-fd8ccff4c-jtrcq\" (UID: \"a7d8f739-da68-4dcc-a17e-47ca4a85f8fe\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq"
Jul 09 23:32:25.896888 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.896729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7d8f739-da68-4dcc-a17e-47ca4a85f8fe-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jtrcq\" (UID: \"a7d8f739-da68-4dcc-a17e-47ca4a85f8fe\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq"
Jul 09 23:32:25.997417 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.997373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdk5z\" (UniqueName: \"kubernetes.io/projected/a7d8f739-da68-4dcc-a17e-47ca4a85f8fe-kube-api-access-rdk5z\") pod \"openshift-lws-operator-fd8ccff4c-jtrcq\" (UID: \"a7d8f739-da68-4dcc-a17e-47ca4a85f8fe\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq"
Jul 09 23:32:25.997604 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.997430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7d8f739-da68-4dcc-a17e-47ca4a85f8fe-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jtrcq\" (UID: \"a7d8f739-da68-4dcc-a17e-47ca4a85f8fe\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq"
Jul 09 23:32:25.997850 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:25.997829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7d8f739-da68-4dcc-a17e-47ca4a85f8fe-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jtrcq\" (UID: \"a7d8f739-da68-4dcc-a17e-47ca4a85f8fe\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq"
Jul 09 23:32:26.006083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:26.006024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdk5z\" (UniqueName: \"kubernetes.io/projected/a7d8f739-da68-4dcc-a17e-47ca4a85f8fe-kube-api-access-rdk5z\") pod \"openshift-lws-operator-fd8ccff4c-jtrcq\" (UID: \"a7d8f739-da68-4dcc-a17e-47ca4a85f8fe\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq"
Jul 09 23:32:26.062887 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:26.062844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq"
Jul 09 23:32:26.184414 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:26.184382    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq"]
Jul 09 23:32:26.187133 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:32:26.187108    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7d8f739_da68_4dcc_a17e_47ca4a85f8fe.slice/crio-db7de2fcb48a057da0b40e46dbc4ed4f80a966080127966b17f29d9fe3ebfcc5 WatchSource:0}: Error finding container db7de2fcb48a057da0b40e46dbc4ed4f80a966080127966b17f29d9fe3ebfcc5: Status 404 returned error can't find the container with id db7de2fcb48a057da0b40e46dbc4ed4f80a966080127966b17f29d9fe3ebfcc5
Jul 09 23:32:26.493903 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:26.493869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq" event={"ID":"a7d8f739-da68-4dcc-a17e-47ca4a85f8fe","Type":"ContainerStarted","Data":"db7de2fcb48a057da0b40e46dbc4ed4f80a966080127966b17f29d9fe3ebfcc5"}
Jul 09 23:32:28.502827 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:28.502743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq" event={"ID":"a7d8f739-da68-4dcc-a17e-47ca4a85f8fe","Type":"ContainerStarted","Data":"a53081876c58df9922185d19737741e6f10eaba7a2b3e2b1acd1c262ee57cc6a"}
Jul 09 23:32:28.518507 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:28.518455    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jtrcq" podStartSLOduration=1.574145996 podStartE2EDuration="3.518439099s" totalImagesPullingTime="1.944293103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:32:28.517362093 +0000 UTC m=+266.661292883" watchObservedRunningTime="2026-07-09 23:32:28.518439099 +0000 UTC m=+266.662369884"
Jul 09 23:32:44.628061 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.628021    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"]
Jul 09 23:32:44.634085 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.634056    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.636711 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.636690    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8zrkh\""
Jul 09 23:32:44.636848 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.636727    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 09 23:32:44.636926 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.636897    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 09 23:32:44.637002 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.636958    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 09 23:32:44.637184 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.637170    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 09 23:32:44.645328 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.645309    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"]
Jul 09 23:32:44.724579 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.724545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-manifests\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.724579 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.724576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.724795 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.724611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbqrx\" (UniqueName: \"kubernetes.io/projected/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-kube-api-access-hbqrx\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.724795 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.724677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-webhook-cert\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.724795 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.724736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-tmp\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.825018 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.824980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbqrx\" (UniqueName: \"kubernetes.io/projected/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-kube-api-access-hbqrx\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.825018 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.825019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-webhook-cert\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.825220 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.825048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-tmp\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.825220 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.825078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-manifests\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.825220 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.825092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.825422 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.825406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-manifests\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.827589 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.827552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-tmp\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.827775 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.827670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.827775 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.827729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-webhook-cert\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.833513 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.833484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbqrx\" (UniqueName: \"kubernetes.io/projected/837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e-kube-api-access-hbqrx\") pod \"opendatahub-operator-controller-manager-c55df6879-nq8k8\" (UID: \"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:44.943685 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:44.943574    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:45.077225 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:45.077199    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"]
Jul 09 23:32:45.079850 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:32:45.079816    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837e94d9_3c7e_473d_ad3d_86b2bf4d8e8e.slice/crio-63c29936f24003dc445fbc16f75bba67f10c04f610cb2f8450816c7b00f09012 WatchSource:0}: Error finding container 63c29936f24003dc445fbc16f75bba67f10c04f610cb2f8450816c7b00f09012: Status 404 returned error can't find the container with id 63c29936f24003dc445fbc16f75bba67f10c04f610cb2f8450816c7b00f09012
Jul 09 23:32:45.565977 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:45.565941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8" event={"ID":"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e","Type":"ContainerStarted","Data":"63c29936f24003dc445fbc16f75bba67f10c04f610cb2f8450816c7b00f09012"}
Jul 09 23:32:49.584855 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:49.584817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8" event={"ID":"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e","Type":"ContainerStarted","Data":"b73a79184aeb80cbe1de13cdd101b044c1f77528d5d7fac6df3b864669274507"}
Jul 09 23:32:50.590794 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:50.590758    2580 generic.go:356] "Generic (PLEG): container finished" podID="837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e" containerID="b73a79184aeb80cbe1de13cdd101b044c1f77528d5d7fac6df3b864669274507" exitCode=0
Jul 09 23:32:50.591187 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:50.590845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8" event={"ID":"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e","Type":"ContainerDied","Data":"b73a79184aeb80cbe1de13cdd101b044c1f77528d5d7fac6df3b864669274507"}
Jul 09 23:32:51.597267 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:51.597237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8" event={"ID":"837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e","Type":"ContainerStarted","Data":"5d0160edc0ae349af092ea354c5188439e1b6d492a7c39db7d13aed133368d23"}
Jul 09 23:32:51.597766 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:51.597311    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:32:51.624393 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:32:51.624338    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8" podStartSLOduration=2.758115086 podStartE2EDuration="7.624320532s" totalImagesPullingTime="3.821501758s" totalInitContainerRuntime="1.044703688s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:32:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:32:51.617747949 +0000 UTC m=+289.761678735" watchObservedRunningTime="2026-07-09 23:32:51.624320532 +0000 UTC m=+289.768251317"
Jul 09 23:33:02.291708 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:02.291678    2580 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 23:33:02.603758 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:02.603674    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-nq8k8"
Jul 09 23:33:04.568125 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.568087    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"]
Jul 09 23:33:04.570227 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.570204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"
Jul 09 23:33:04.572831 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.572803    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 09 23:33:04.572944 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.572807    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 23:33:04.572944 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.572813    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 09 23:33:04.573806 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.573790    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 23:33:04.586720 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.586694    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"]
Jul 09 23:33:04.673089 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.673060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxhh9\" (UniqueName: \"kubernetes.io/projected/bf33e8b0-4ed1-4c59-ba89-0493fa839f33-kube-api-access-fxhh9\") pod \"kube-auth-proxy-6dff7df579-8sc5p\" (UID: \"bf33e8b0-4ed1-4c59-ba89-0493fa839f33\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"
Jul 09 23:33:04.673252 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.673103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf33e8b0-4ed1-4c59-ba89-0493fa839f33-tmp\") pod \"kube-auth-proxy-6dff7df579-8sc5p\" (UID: \"bf33e8b0-4ed1-4c59-ba89-0493fa839f33\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"
Jul 09 23:33:04.673252 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.673137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf33e8b0-4ed1-4c59-ba89-0493fa839f33-tls-certs\") pod \"kube-auth-proxy-6dff7df579-8sc5p\" (UID: \"bf33e8b0-4ed1-4c59-ba89-0493fa839f33\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"
Jul 09 23:33:04.773917 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.773884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhh9\" (UniqueName: \"kubernetes.io/projected/bf33e8b0-4ed1-4c59-ba89-0493fa839f33-kube-api-access-fxhh9\") pod \"kube-auth-proxy-6dff7df579-8sc5p\" (UID: \"bf33e8b0-4ed1-4c59-ba89-0493fa839f33\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"
Jul 09 23:33:04.773917 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.773928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf33e8b0-4ed1-4c59-ba89-0493fa839f33-tmp\") pod \"kube-auth-proxy-6dff7df579-8sc5p\" (UID: \"bf33e8b0-4ed1-4c59-ba89-0493fa839f33\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"
Jul 09 23:33:04.774139 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.773953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf33e8b0-4ed1-4c59-ba89-0493fa839f33-tls-certs\") pod \"kube-auth-proxy-6dff7df579-8sc5p\" (UID: \"bf33e8b0-4ed1-4c59-ba89-0493fa839f33\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"
Jul 09 23:33:04.776252 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.776218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf33e8b0-4ed1-4c59-ba89-0493fa839f33-tmp\") pod \"kube-auth-proxy-6dff7df579-8sc5p\" (UID: \"bf33e8b0-4ed1-4c59-ba89-0493fa839f33\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"
Jul 09 23:33:04.776440 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.776420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf33e8b0-4ed1-4c59-ba89-0493fa839f33-tls-certs\") pod \"kube-auth-proxy-6dff7df579-8sc5p\" (UID: \"bf33e8b0-4ed1-4c59-ba89-0493fa839f33\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"
Jul 09 23:33:04.784399 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.784372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhh9\" (UniqueName: \"kubernetes.io/projected/bf33e8b0-4ed1-4c59-ba89-0493fa839f33-kube-api-access-fxhh9\") pod \"kube-auth-proxy-6dff7df579-8sc5p\" (UID: \"bf33e8b0-4ed1-4c59-ba89-0493fa839f33\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"
Jul 09 23:33:04.879557 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.879466    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"
Jul 09 23:33:04.910204 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.910172    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8hbx9"]
Jul 09 23:33:04.912408 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.912392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9"
Jul 09 23:33:04.914633 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.914605    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 09 23:33:04.914787 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.914732    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7pbbv\""
Jul 09 23:33:04.923967 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.923938    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8hbx9"]
Jul 09 23:33:04.988722 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.984992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg4n5\" (UniqueName: \"kubernetes.io/projected/a69d3267-69d9-47eb-b954-e0286f45fe6a-kube-api-access-tg4n5\") pod \"odh-model-controller-7998c67dc4-8hbx9\" (UID: \"a69d3267-69d9-47eb-b954-e0286f45fe6a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9"
Jul 09 23:33:04.988722 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:04.985110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a69d3267-69d9-47eb-b954-e0286f45fe6a-cert\") pod \"odh-model-controller-7998c67dc4-8hbx9\" (UID: \"a69d3267-69d9-47eb-b954-e0286f45fe6a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9"
Jul 09 23:33:05.055016 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.054835    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p"]
Jul 09 23:33:05.057766 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:33:05.057734    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf33e8b0_4ed1_4c59_ba89_0493fa839f33.slice/crio-65ab164ec570d0c4fa347dec421cf1ad30ca08fa3168b8365ad8562634166e66 WatchSource:0}: Error finding container 65ab164ec570d0c4fa347dec421cf1ad30ca08fa3168b8365ad8562634166e66: Status 404 returned error can't find the container with id 65ab164ec570d0c4fa347dec421cf1ad30ca08fa3168b8365ad8562634166e66
Jul 09 23:33:05.060004 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.059986    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 23:33:05.086238 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.086204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a69d3267-69d9-47eb-b954-e0286f45fe6a-cert\") pod \"odh-model-controller-7998c67dc4-8hbx9\" (UID: \"a69d3267-69d9-47eb-b954-e0286f45fe6a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9"
Jul 09 23:33:05.086238 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.086243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg4n5\" (UniqueName: \"kubernetes.io/projected/a69d3267-69d9-47eb-b954-e0286f45fe6a-kube-api-access-tg4n5\") pod \"odh-model-controller-7998c67dc4-8hbx9\" (UID: \"a69d3267-69d9-47eb-b954-e0286f45fe6a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9"
Jul 09 23:33:05.086448 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:33:05.086364    2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 09 23:33:05.086448 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:33:05.086440    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a69d3267-69d9-47eb-b954-e0286f45fe6a-cert podName:a69d3267-69d9-47eb-b954-e0286f45fe6a nodeName:}" failed. No retries permitted until 2026-07-09 23:33:05.586419076 +0000 UTC m=+303.730349854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a69d3267-69d9-47eb-b954-e0286f45fe6a-cert") pod "odh-model-controller-7998c67dc4-8hbx9" (UID: "a69d3267-69d9-47eb-b954-e0286f45fe6a") : secret "odh-model-controller-webhook-cert" not found
Jul 09 23:33:05.096667 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.096628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg4n5\" (UniqueName: \"kubernetes.io/projected/a69d3267-69d9-47eb-b954-e0286f45fe6a-kube-api-access-tg4n5\") pod \"odh-model-controller-7998c67dc4-8hbx9\" (UID: \"a69d3267-69d9-47eb-b954-e0286f45fe6a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9"
Jul 09 23:33:05.589785 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.589749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a69d3267-69d9-47eb-b954-e0286f45fe6a-cert\") pod \"odh-model-controller-7998c67dc4-8hbx9\" (UID: \"a69d3267-69d9-47eb-b954-e0286f45fe6a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9"
Jul 09 23:33:05.592070 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.592049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a69d3267-69d9-47eb-b954-e0286f45fe6a-cert\") pod \"odh-model-controller-7998c67dc4-8hbx9\" (UID: \"a69d3267-69d9-47eb-b954-e0286f45fe6a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9"
Jul 09 23:33:05.656851 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.656817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p" event={"ID":"bf33e8b0-4ed1-4c59-ba89-0493fa839f33","Type":"ContainerStarted","Data":"65ab164ec570d0c4fa347dec421cf1ad30ca08fa3168b8365ad8562634166e66"}
Jul 09 23:33:05.825061 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.825014    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9"
Jul 09 23:33:05.852311 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.851951    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6hk6p"]
Jul 09 23:33:05.854743 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.854405    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p"
Jul 09 23:33:05.856870 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.856846    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-q8bjq\""
Jul 09 23:33:05.857057 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.857038    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 09 23:33:05.863913 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.863858    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6hk6p"]
Jul 09 23:33:05.892150 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.891985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6065bde9-3c70-46c7-8c49-f283ee15ecb0-tls-certs\") pod \"model-serving-api-54d89bd5fc-6hk6p\" (UID: \"6065bde9-3c70-46c7-8c49-f283ee15ecb0\") " pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p"
Jul 09 23:33:05.892150 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.892045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj9p9\" (UniqueName: \"kubernetes.io/projected/6065bde9-3c70-46c7-8c49-f283ee15ecb0-kube-api-access-fj9p9\") pod \"model-serving-api-54d89bd5fc-6hk6p\" (UID: \"6065bde9-3c70-46c7-8c49-f283ee15ecb0\") " pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p"
Jul 09 23:33:05.894716 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.894687    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"]
Jul 09 23:33:05.899357 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.899334    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:05.901918 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.901896    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 09 23:33:05.902506 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.902489    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 09 23:33:05.902627 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.902613    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9ctd9\""
Jul 09 23:33:05.902992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.902871    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 09 23:33:05.902992 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.902883    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 23:33:05.910934 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.910907    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"]
Jul 09 23:33:05.985342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.985307    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8hbx9"]
Jul 09 23:33:05.990810 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:33:05.990777    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69d3267_69d9_47eb_b954_e0286f45fe6a.slice/crio-006869a47eb62d4e44c999fde2aa189301d9d1e0099e073d4ce7f6938908746c WatchSource:0}: Error finding container 006869a47eb62d4e44c999fde2aa189301d9d1e0099e073d4ce7f6938908746c: Status 404 returned error can't find the container with id 006869a47eb62d4e44c999fde2aa189301d9d1e0099e073d4ce7f6938908746c
Jul 09 23:33:05.992514 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.992487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:05.992623 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.992550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-token\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:05.992623 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.992576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f85dc\" (UniqueName: \"kubernetes.io/projected/7eff4495-2e18-4b9b-b402-607c5df3de94-kube-api-access-f85dc\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:05.992623 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.992602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eff4495-2e18-4b9b-b402-607c5df3de94-local-certs\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:05.992820 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.992625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:05.992820 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.992745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6065bde9-3c70-46c7-8c49-f283ee15ecb0-tls-certs\") pod \"model-serving-api-54d89bd5fc-6hk6p\" (UID: \"6065bde9-3c70-46c7-8c49-f283ee15ecb0\") " pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p"
Jul 09 23:33:05.992820 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.992790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9p9\" (UniqueName: \"kubernetes.io/projected/6065bde9-3c70-46c7-8c49-f283ee15ecb0-kube-api-access-fj9p9\") pod \"model-serving-api-54d89bd5fc-6hk6p\" (UID: \"6065bde9-3c70-46c7-8c49-f283ee15ecb0\") " pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p"
Jul 09 23:33:05.992967 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.992846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:05.992967 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:05.992872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-cacerts\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:05.992967 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:33:05.992894    2580 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 09 23:33:05.992967 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:33:05.992947    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6065bde9-3c70-46c7-8c49-f283ee15ecb0-tls-certs podName:6065bde9-3c70-46c7-8c49-f283ee15ecb0 nodeName:}" failed. No retries permitted until 2026-07-09 23:33:06.492928601 +0000 UTC m=+304.636859371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6065bde9-3c70-46c7-8c49-f283ee15ecb0-tls-certs") pod "model-serving-api-54d89bd5fc-6hk6p" (UID: "6065bde9-3c70-46c7-8c49-f283ee15ecb0") : secret "model-serving-api-tls" not found
Jul 09 23:33:06.002008 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.001979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9p9\" (UniqueName: \"kubernetes.io/projected/6065bde9-3c70-46c7-8c49-f283ee15ecb0-kube-api-access-fj9p9\") pod \"model-serving-api-54d89bd5fc-6hk6p\" (UID: \"6065bde9-3c70-46c7-8c49-f283ee15ecb0\") " pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p"
Jul 09 23:33:06.093950 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.093910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.093950 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.093952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-cacerts\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.094194 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.093978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.094194 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.094099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-token\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.094194 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.094139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f85dc\" (UniqueName: \"kubernetes.io/projected/7eff4495-2e18-4b9b-b402-607c5df3de94-kube-api-access-f85dc\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.094194 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.094163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eff4495-2e18-4b9b-b402-607c5df3de94-local-certs\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.094194 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.094188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.094920 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.094885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.096423 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.096406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.096501 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.096437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.096617 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.096604    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eff4495-2e18-4b9b-b402-607c5df3de94-local-certs\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.096785 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.096758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-cacerts\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.101369 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.101334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-token\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.101544 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.101525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f85dc\" (UniqueName: \"kubernetes.io/projected/7eff4495-2e18-4b9b-b402-607c5df3de94-kube-api-access-f85dc\") pod \"istiod-openshift-gateway-7574459656-rwnz8\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.211168 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.211074    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:06.380980 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.380955    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"]
Jul 09 23:33:06.384000 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:33:06.383967    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eff4495_2e18_4b9b_b402_607c5df3de94.slice/crio-088f57679bd08216ffa93c96ac56c23844840d2b242325d1fc90fbcfc900e557 WatchSource:0}: Error finding container 088f57679bd08216ffa93c96ac56c23844840d2b242325d1fc90fbcfc900e557: Status 404 returned error can't find the container with id 088f57679bd08216ffa93c96ac56c23844840d2b242325d1fc90fbcfc900e557
Jul 09 23:33:06.497670 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.497559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6065bde9-3c70-46c7-8c49-f283ee15ecb0-tls-certs\") pod \"model-serving-api-54d89bd5fc-6hk6p\" (UID: \"6065bde9-3c70-46c7-8c49-f283ee15ecb0\") " pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p"
Jul 09 23:33:06.500393 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.500369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6065bde9-3c70-46c7-8c49-f283ee15ecb0-tls-certs\") pod \"model-serving-api-54d89bd5fc-6hk6p\" (UID: \"6065bde9-3c70-46c7-8c49-f283ee15ecb0\") " pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p"
Jul 09 23:33:06.662433 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.662398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9" event={"ID":"a69d3267-69d9-47eb-b954-e0286f45fe6a","Type":"ContainerStarted","Data":"006869a47eb62d4e44c999fde2aa189301d9d1e0099e073d4ce7f6938908746c"}
Jul 09 23:33:06.663529 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.663506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8" event={"ID":"7eff4495-2e18-4b9b-b402-607c5df3de94","Type":"ContainerStarted","Data":"088f57679bd08216ffa93c96ac56c23844840d2b242325d1fc90fbcfc900e557"}
Jul 09 23:33:06.768828 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.768790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p"
Jul 09 23:33:06.941531 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:06.941506    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6hk6p"]
Jul 09 23:33:06.944474 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:33:06.944440    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6065bde9_3c70_46c7_8c49_f283ee15ecb0.slice/crio-802b467d1ac621ad1bd4a0b63c52f22b88b663d7ffe2c7dfee36fd93d83657ec WatchSource:0}: Error finding container 802b467d1ac621ad1bd4a0b63c52f22b88b663d7ffe2c7dfee36fd93d83657ec: Status 404 returned error can't find the container with id 802b467d1ac621ad1bd4a0b63c52f22b88b663d7ffe2c7dfee36fd93d83657ec
Jul 09 23:33:07.671098 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:07.671044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p" event={"ID":"6065bde9-3c70-46c7-8c49-f283ee15ecb0","Type":"ContainerStarted","Data":"802b467d1ac621ad1bd4a0b63c52f22b88b663d7ffe2c7dfee36fd93d83657ec"}
Jul 09 23:33:10.840973 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:10.840923    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:33:10.841409 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:10.841012    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:33:11.693322 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:11.693279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9" event={"ID":"a69d3267-69d9-47eb-b954-e0286f45fe6a","Type":"ContainerStarted","Data":"67022221f4603196f092cd065c66851e971dddcfb89329cb71a0c496b20a6791"}
Jul 09 23:33:11.693504 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:11.693395    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9"
Jul 09 23:33:11.694993 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:11.694963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p" event={"ID":"6065bde9-3c70-46c7-8c49-f283ee15ecb0","Type":"ContainerStarted","Data":"b693488237e7c85215200f91e58bf6d2e727600a146aab4a33174f0613696bcb"}
Jul 09 23:33:11.695136 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:11.695036    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p"
Jul 09 23:33:11.696785 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:11.696757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8" event={"ID":"7eff4495-2e18-4b9b-b402-607c5df3de94","Type":"ContainerStarted","Data":"22cfd433cd2a78aab468be7eb1e38b5a6e61b9b76883ecd09eccee9f5c929717"}
Jul 09 23:33:11.696995 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:11.696957    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:11.698552 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:11.698528    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-rwnz8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 09 23:33:11.698643 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:11.698575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p" event={"ID":"bf33e8b0-4ed1-4c59-ba89-0493fa839f33","Type":"ContainerStarted","Data":"6bbd40ea63a2337fbf73fa169501f5524bb4c2fdc3c675b67fc2ac7204de5277"}
Jul 09 23:33:11.698643 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:11.698575    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8" podUID="7eff4495-2e18-4b9b-b402-607c5df3de94" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 23:33:11.709092 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:11.709054    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9" podStartSLOduration=2.886098047 podStartE2EDuration="7.709041275s" totalImagesPullingTime="4.822943228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:33:11.707713964 +0000 UTC m=+309.851644752" watchObservedRunningTime="2026-07-09 23:33:11.709041275 +0000 UTC m=+309.852972038"
Jul 09 23:33:11.723613 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:11.723567    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6dff7df579-8sc5p" podStartSLOduration=1.9742843319999999 podStartE2EDuration="7.723552632s" totalImagesPullingTime="5.7492683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:33:11.722496886 +0000 UTC m=+309.866427673" watchObservedRunningTime="2026-07-09 23:33:11.723552632 +0000 UTC m=+309.867483419"
Jul 09 23:33:11.741810 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:11.741753    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p" podStartSLOduration=2.84256403 podStartE2EDuration="6.741733976s" totalImagesPullingTime="3.899169946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:33:11.740083005 +0000 UTC m=+309.884013790" watchObservedRunningTime="2026-07-09 23:33:11.741733976 +0000 UTC m=+309.885664763"
Jul 09 23:33:11.761037 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:11.760989    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8" podStartSLOduration=2.3063742 podStartE2EDuration="6.760972424s" totalImagesPullingTime="4.454598224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:33:11.758806794 +0000 UTC m=+309.902737581" watchObservedRunningTime="2026-07-09 23:33:11.760972424 +0000 UTC m=+309.904903209"
Jul 09 23:33:12.703240 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:12.703201    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:14.607114 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.607078    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"]
Jul 09 23:33:14.610989 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.610970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:33:14.613755 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.613730    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 09 23:33:14.614747 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.614727    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wmp6x\""
Jul 09 23:33:14.625320 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.625294    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"]
Jul 09 23:33:14.627331 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.627307    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"]
Jul 09 23:33:14.630819 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.630802    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"
Jul 09 23:33:14.633253 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.633234    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 09 23:33:14.633253 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.633248    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-r66t2\""
Jul 09 23:33:14.641818 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.641794    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"]
Jul 09 23:33:14.678709 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.678675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686df5b5-af27-4125-8c70-5b8f0a52afdf-cert\") pod \"kserve-controller-manager-d6f4b65b8-l9vqc\" (UID: \"686df5b5-af27-4125-8c70-5b8f0a52afdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:33:14.678900 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.678729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrgxp\" (UniqueName: \"kubernetes.io/projected/686df5b5-af27-4125-8c70-5b8f0a52afdf-kube-api-access-mrgxp\") pod \"kserve-controller-manager-d6f4b65b8-l9vqc\" (UID: \"686df5b5-af27-4125-8c70-5b8f0a52afdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:33:14.779965 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.779927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0719fe1b-21a5-4adc-b802-cd591b01cc17-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-nmfvk\" (UID: \"0719fe1b-21a5-4adc-b802-cd591b01cc17\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"
Jul 09 23:33:14.780153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.779970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686df5b5-af27-4125-8c70-5b8f0a52afdf-cert\") pod \"kserve-controller-manager-d6f4b65b8-l9vqc\" (UID: \"686df5b5-af27-4125-8c70-5b8f0a52afdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:33:14.780153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.779994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrgxp\" (UniqueName: \"kubernetes.io/projected/686df5b5-af27-4125-8c70-5b8f0a52afdf-kube-api-access-mrgxp\") pod \"kserve-controller-manager-d6f4b65b8-l9vqc\" (UID: \"686df5b5-af27-4125-8c70-5b8f0a52afdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:33:14.780153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.780078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hbv7\" (UniqueName: \"kubernetes.io/projected/0719fe1b-21a5-4adc-b802-cd591b01cc17-kube-api-access-8hbv7\") pod \"llmisvc-controller-manager-777c6bd7d5-nmfvk\" (UID: \"0719fe1b-21a5-4adc-b802-cd591b01cc17\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"
Jul 09 23:33:14.780153 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:33:14.780110    2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 09 23:33:14.780356 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:33:14.780175    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686df5b5-af27-4125-8c70-5b8f0a52afdf-cert podName:686df5b5-af27-4125-8c70-5b8f0a52afdf nodeName:}" failed. No retries permitted until 2026-07-09 23:33:15.280155199 +0000 UTC m=+313.424085967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/686df5b5-af27-4125-8c70-5b8f0a52afdf-cert") pod "kserve-controller-manager-d6f4b65b8-l9vqc" (UID: "686df5b5-af27-4125-8c70-5b8f0a52afdf") : secret "kserve-webhook-server-cert" not found
Jul 09 23:33:14.795000 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.794969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrgxp\" (UniqueName: \"kubernetes.io/projected/686df5b5-af27-4125-8c70-5b8f0a52afdf-kube-api-access-mrgxp\") pod \"kserve-controller-manager-d6f4b65b8-l9vqc\" (UID: \"686df5b5-af27-4125-8c70-5b8f0a52afdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:33:14.881347 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.881244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0719fe1b-21a5-4adc-b802-cd591b01cc17-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-nmfvk\" (UID: \"0719fe1b-21a5-4adc-b802-cd591b01cc17\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"
Jul 09 23:33:14.881347 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.881308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hbv7\" (UniqueName: \"kubernetes.io/projected/0719fe1b-21a5-4adc-b802-cd591b01cc17-kube-api-access-8hbv7\") pod \"llmisvc-controller-manager-777c6bd7d5-nmfvk\" (UID: \"0719fe1b-21a5-4adc-b802-cd591b01cc17\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"
Jul 09 23:33:14.881581 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:33:14.881396    2580 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 09 23:33:14.881581 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:33:14.881456    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0719fe1b-21a5-4adc-b802-cd591b01cc17-cert podName:0719fe1b-21a5-4adc-b802-cd591b01cc17 nodeName:}" failed. No retries permitted until 2026-07-09 23:33:15.381441049 +0000 UTC m=+313.525371813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0719fe1b-21a5-4adc-b802-cd591b01cc17-cert") pod "llmisvc-controller-manager-777c6bd7d5-nmfvk" (UID: "0719fe1b-21a5-4adc-b802-cd591b01cc17") : secret "llmisvc-webhook-server-cert" not found
Jul 09 23:33:14.890829 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:14.890797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hbv7\" (UniqueName: \"kubernetes.io/projected/0719fe1b-21a5-4adc-b802-cd591b01cc17-kube-api-access-8hbv7\") pod \"llmisvc-controller-manager-777c6bd7d5-nmfvk\" (UID: \"0719fe1b-21a5-4adc-b802-cd591b01cc17\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"
Jul 09 23:33:15.284985 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:15.284946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686df5b5-af27-4125-8c70-5b8f0a52afdf-cert\") pod \"kserve-controller-manager-d6f4b65b8-l9vqc\" (UID: \"686df5b5-af27-4125-8c70-5b8f0a52afdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:33:15.287305 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:15.287275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686df5b5-af27-4125-8c70-5b8f0a52afdf-cert\") pod \"kserve-controller-manager-d6f4b65b8-l9vqc\" (UID: \"686df5b5-af27-4125-8c70-5b8f0a52afdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:33:15.386094 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:15.386047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0719fe1b-21a5-4adc-b802-cd591b01cc17-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-nmfvk\" (UID: \"0719fe1b-21a5-4adc-b802-cd591b01cc17\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"
Jul 09 23:33:15.388442 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:15.388420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0719fe1b-21a5-4adc-b802-cd591b01cc17-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-nmfvk\" (UID: \"0719fe1b-21a5-4adc-b802-cd591b01cc17\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"
Jul 09 23:33:15.522506 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:15.522467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:33:15.544828 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:15.544762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"
Jul 09 23:33:15.675025 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:15.675002    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"]
Jul 09 23:33:15.676768 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:33:15.676743    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686df5b5_af27_4125_8c70_5b8f0a52afdf.slice/crio-38e2d5fbdef90da36cdc2f683d31082bd8cf84dbb6c78c8adad1ac31d4f2de63 WatchSource:0}: Error finding container 38e2d5fbdef90da36cdc2f683d31082bd8cf84dbb6c78c8adad1ac31d4f2de63: Status 404 returned error can't find the container with id 38e2d5fbdef90da36cdc2f683d31082bd8cf84dbb6c78c8adad1ac31d4f2de63
Jul 09 23:33:15.696470 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:15.696444    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"]
Jul 09 23:33:15.698741 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:33:15.698710    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0719fe1b_21a5_4adc_b802_cd591b01cc17.slice/crio-6cdad17623175eeeab74c16c748f34c4baa2d0d520e529eb70fcf4c473346f86 WatchSource:0}: Error finding container 6cdad17623175eeeab74c16c748f34c4baa2d0d520e529eb70fcf4c473346f86: Status 404 returned error can't find the container with id 6cdad17623175eeeab74c16c748f34c4baa2d0d520e529eb70fcf4c473346f86
Jul 09 23:33:15.715881 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:15.715848    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk" event={"ID":"0719fe1b-21a5-4adc-b802-cd591b01cc17","Type":"ContainerStarted","Data":"6cdad17623175eeeab74c16c748f34c4baa2d0d520e529eb70fcf4c473346f86"}
Jul 09 23:33:15.716910 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:15.716887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc" event={"ID":"686df5b5-af27-4125-8c70-5b8f0a52afdf","Type":"ContainerStarted","Data":"38e2d5fbdef90da36cdc2f683d31082bd8cf84dbb6c78c8adad1ac31d4f2de63"}
Jul 09 23:33:17.293246 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.293216    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"]
Jul 09 23:33:17.293681 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.293381    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8" podUID="7eff4495-2e18-4b9b-b402-607c5df3de94" containerName="discovery" containerID="cri-o://22cfd433cd2a78aab468be7eb1e38b5a6e61b9b76883ecd09eccee9f5c929717" gracePeriod=30
Jul 09 23:33:17.575964 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.575934    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:17.707947 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.707915    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-cacerts\") pod \"7eff4495-2e18-4b9b-b402-607c5df3de94\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") "
Jul 09 23:33:17.707947 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.707951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-token\") pod \"7eff4495-2e18-4b9b-b402-607c5df3de94\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") "
Jul 09 23:33:17.708173 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.707976    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-kubeconfig\") pod \"7eff4495-2e18-4b9b-b402-607c5df3de94\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") "
Jul 09 23:33:17.708173 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.708005    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-csr-ca-configmap\") pod \"7eff4495-2e18-4b9b-b402-607c5df3de94\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") "
Jul 09 23:33:17.708173 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.708042    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f85dc\" (UniqueName: \"kubernetes.io/projected/7eff4495-2e18-4b9b-b402-607c5df3de94-kube-api-access-f85dc\") pod \"7eff4495-2e18-4b9b-b402-607c5df3de94\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") "
Jul 09 23:33:17.708173 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.708106    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-csr-dns-cert\") pod \"7eff4495-2e18-4b9b-b402-607c5df3de94\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") "
Jul 09 23:33:17.708173 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.708168    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eff4495-2e18-4b9b-b402-607c5df3de94-local-certs\") pod \"7eff4495-2e18-4b9b-b402-607c5df3de94\" (UID: \"7eff4495-2e18-4b9b-b402-607c5df3de94\") "
Jul 09 23:33:17.708778 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.708744    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7eff4495-2e18-4b9b-b402-607c5df3de94" (UID: "7eff4495-2e18-4b9b-b402-607c5df3de94"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 23:33:17.710782 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.710755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7eff4495-2e18-4b9b-b402-607c5df3de94" (UID: "7eff4495-2e18-4b9b-b402-607c5df3de94"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 23:33:17.710981 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.710891    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eff4495-2e18-4b9b-b402-607c5df3de94-local-certs" (OuterVolumeSpecName: "local-certs") pod "7eff4495-2e18-4b9b-b402-607c5df3de94" (UID: "7eff4495-2e18-4b9b-b402-607c5df3de94"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:33:17.710981 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.710907    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-cacerts" (OuterVolumeSpecName: "cacerts") pod "7eff4495-2e18-4b9b-b402-607c5df3de94" (UID: "7eff4495-2e18-4b9b-b402-607c5df3de94"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 23:33:17.710981 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.710916    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-token" (OuterVolumeSpecName: "istio-token") pod "7eff4495-2e18-4b9b-b402-607c5df3de94" (UID: "7eff4495-2e18-4b9b-b402-607c5df3de94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:33:17.710981 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.710972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7eff4495-2e18-4b9b-b402-607c5df3de94" (UID: "7eff4495-2e18-4b9b-b402-607c5df3de94"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 23:33:17.711225 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.711174    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eff4495-2e18-4b9b-b402-607c5df3de94-kube-api-access-f85dc" (OuterVolumeSpecName: "kube-api-access-f85dc") pod "7eff4495-2e18-4b9b-b402-607c5df3de94" (UID: "7eff4495-2e18-4b9b-b402-607c5df3de94"). InnerVolumeSpecName "kube-api-access-f85dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:33:17.728034 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.728001    2580 generic.go:356] "Generic (PLEG): container finished" podID="7eff4495-2e18-4b9b-b402-607c5df3de94" containerID="22cfd433cd2a78aab468be7eb1e38b5a6e61b9b76883ecd09eccee9f5c929717" exitCode=0
Jul 09 23:33:17.728158 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.728060    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"
Jul 09 23:33:17.728158 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.728087    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8" event={"ID":"7eff4495-2e18-4b9b-b402-607c5df3de94","Type":"ContainerDied","Data":"22cfd433cd2a78aab468be7eb1e38b5a6e61b9b76883ecd09eccee9f5c929717"}
Jul 09 23:33:17.728158 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.728142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8" event={"ID":"7eff4495-2e18-4b9b-b402-607c5df3de94","Type":"ContainerDied","Data":"088f57679bd08216ffa93c96ac56c23844840d2b242325d1fc90fbcfc900e557"}
Jul 09 23:33:17.728284 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.728163    2580 scope.go:117] "RemoveContainer" containerID="22cfd433cd2a78aab468be7eb1e38b5a6e61b9b76883ecd09eccee9f5c929717"
Jul 09 23:33:17.750785 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.750763    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"]
Jul 09 23:33:17.755358 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.755328    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rwnz8"]
Jul 09 23:33:17.809120 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.809001    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-csr-dns-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:33:17.809120 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.809034    2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eff4495-2e18-4b9b-b402-607c5df3de94-local-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:33:17.809120 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.809050    2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-cacerts\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:33:17.809120 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.809062    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-token\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:33:17.809120 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.809074    2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-kubeconfig\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:33:17.809120 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.809085    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eff4495-2e18-4b9b-b402-607c5df3de94-istio-csr-ca-configmap\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:33:17.809120 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:17.809097    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f85dc\" (UniqueName: \"kubernetes.io/projected/7eff4495-2e18-4b9b-b402-607c5df3de94-kube-api-access-f85dc\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:33:18.217421 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:18.217381    2580 scope.go:117] "RemoveContainer" containerID="22cfd433cd2a78aab468be7eb1e38b5a6e61b9b76883ecd09eccee9f5c929717"
Jul 09 23:33:18.217797 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:33:18.217759    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22cfd433cd2a78aab468be7eb1e38b5a6e61b9b76883ecd09eccee9f5c929717\": container with ID starting with 22cfd433cd2a78aab468be7eb1e38b5a6e61b9b76883ecd09eccee9f5c929717 not found: ID does not exist" containerID="22cfd433cd2a78aab468be7eb1e38b5a6e61b9b76883ecd09eccee9f5c929717"
Jul 09 23:33:18.217908 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:18.217807    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22cfd433cd2a78aab468be7eb1e38b5a6e61b9b76883ecd09eccee9f5c929717"} err="failed to get container status \"22cfd433cd2a78aab468be7eb1e38b5a6e61b9b76883ecd09eccee9f5c929717\": rpc error: code = NotFound desc = could not find container \"22cfd433cd2a78aab468be7eb1e38b5a6e61b9b76883ecd09eccee9f5c929717\": container with ID starting with 22cfd433cd2a78aab468be7eb1e38b5a6e61b9b76883ecd09eccee9f5c929717 not found: ID does not exist"
Jul 09 23:33:18.419752 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:18.419716    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eff4495-2e18-4b9b-b402-607c5df3de94" path="/var/lib/kubelet/pods/7eff4495-2e18-4b9b-b402-607c5df3de94/volumes"
Jul 09 23:33:19.739603 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:19.739557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk" event={"ID":"0719fe1b-21a5-4adc-b802-cd591b01cc17","Type":"ContainerStarted","Data":"c9fb547c8978c4011eb0632da00e154c07790dfe908c4400b3c700f9f9c453b9"}
Jul 09 23:33:19.740094 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:19.739675    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"
Jul 09 23:33:19.741147 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:19.741118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc" event={"ID":"686df5b5-af27-4125-8c70-5b8f0a52afdf","Type":"ContainerStarted","Data":"65cf6163eedcc3c3c81c41343bb2cbf1edf84bef5cba736e6bc3e7734ba9268e"}
Jul 09 23:33:19.741243 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:19.741178    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:33:19.757782 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:19.757731    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk" podStartSLOduration=2.595211415 podStartE2EDuration="5.757717112s" totalImagesPullingTime="3.162505697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:33:19.755842108 +0000 UTC m=+317.899772907" watchObservedRunningTime="2026-07-09 23:33:19.757717112 +0000 UTC m=+317.901647897"
Jul 09 23:33:19.771725 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:19.771686    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc" podStartSLOduration=3.177488594 podStartE2EDuration="5.771672049s" totalImagesPullingTime="2.594183455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:33:19.770429235 +0000 UTC m=+317.914360021" watchObservedRunningTime="2026-07-09 23:33:19.771672049 +0000 UTC m=+317.915602833"
Jul 09 23:33:22.704881 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:22.704849    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8hbx9"
Jul 09 23:33:22.706827 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:22.706806    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-6hk6p"
Jul 09 23:33:50.747461 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:50.747425    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nmfvk"
Jul 09 23:33:50.750515 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:33:50.750493    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:34:05.642107 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:05.642071    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d2wcm"]
Jul 09 23:34:05.642502 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:05.642394    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eff4495-2e18-4b9b-b402-607c5df3de94" containerName="discovery"
Jul 09 23:34:05.642502 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:05.642405    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eff4495-2e18-4b9b-b402-607c5df3de94" containerName="discovery"
Jul 09 23:34:05.642502 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:05.642460    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eff4495-2e18-4b9b-b402-607c5df3de94" containerName="discovery"
Jul 09 23:34:05.645979 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:05.645961    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d2wcm"
Jul 09 23:34:05.648449 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:05.648426    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 23:34:05.649566 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:05.649534    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-vt7fg\""
Jul 09 23:34:05.649566 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:05.649550    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 23:34:05.653465 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:05.653441    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d2wcm"]
Jul 09 23:34:05.782065 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:05.782028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wq98\" (UniqueName: \"kubernetes.io/projected/08ac9627-256d-4abf-8247-4825a3a4cf5f-kube-api-access-9wq98\") pod \"kuadrant-operator-catalog-d2wcm\" (UID: \"08ac9627-256d-4abf-8247-4825a3a4cf5f\") " pod="kuadrant-system/kuadrant-operator-catalog-d2wcm"
Jul 09 23:34:05.883114 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:05.883067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wq98\" (UniqueName: \"kubernetes.io/projected/08ac9627-256d-4abf-8247-4825a3a4cf5f-kube-api-access-9wq98\") pod \"kuadrant-operator-catalog-d2wcm\" (UID: \"08ac9627-256d-4abf-8247-4825a3a4cf5f\") " pod="kuadrant-system/kuadrant-operator-catalog-d2wcm"
Jul 09 23:34:05.891304 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:05.891276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wq98\" (UniqueName: \"kubernetes.io/projected/08ac9627-256d-4abf-8247-4825a3a4cf5f-kube-api-access-9wq98\") pod \"kuadrant-operator-catalog-d2wcm\" (UID: \"08ac9627-256d-4abf-8247-4825a3a4cf5f\") " pod="kuadrant-system/kuadrant-operator-catalog-d2wcm"
Jul 09 23:34:05.957541 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:05.957450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d2wcm"
Jul 09 23:34:06.007037 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:06.007002    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d2wcm"]
Jul 09 23:34:06.108159 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:06.108108    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d2wcm"]
Jul 09 23:34:06.110752 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:34:06.110725    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ac9627_256d_4abf_8247_4825a3a4cf5f.slice/crio-4c05fd217e5ac73effbfd9c09e92a3b8a37eba29d83e9b2e86a0df0ca5c29387 WatchSource:0}: Error finding container 4c05fd217e5ac73effbfd9c09e92a3b8a37eba29d83e9b2e86a0df0ca5c29387: Status 404 returned error can't find the container with id 4c05fd217e5ac73effbfd9c09e92a3b8a37eba29d83e9b2e86a0df0ca5c29387
Jul 09 23:34:06.215044 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:06.214961    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tf7wp"]
Jul 09 23:34:06.218984 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:06.218964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tf7wp"
Jul 09 23:34:06.225922 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:06.225668    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tf7wp"]
Jul 09 23:34:06.286784 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:06.286745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hjqx\" (UniqueName: \"kubernetes.io/projected/f7f87602-3ecb-463c-9956-2af87f8fffb0-kube-api-access-9hjqx\") pod \"kuadrant-operator-catalog-tf7wp\" (UID: \"f7f87602-3ecb-463c-9956-2af87f8fffb0\") " pod="kuadrant-system/kuadrant-operator-catalog-tf7wp"
Jul 09 23:34:06.388043 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:06.388000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hjqx\" (UniqueName: \"kubernetes.io/projected/f7f87602-3ecb-463c-9956-2af87f8fffb0-kube-api-access-9hjqx\") pod \"kuadrant-operator-catalog-tf7wp\" (UID: \"f7f87602-3ecb-463c-9956-2af87f8fffb0\") " pod="kuadrant-system/kuadrant-operator-catalog-tf7wp"
Jul 09 23:34:06.395490 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:06.395454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hjqx\" (UniqueName: \"kubernetes.io/projected/f7f87602-3ecb-463c-9956-2af87f8fffb0-kube-api-access-9hjqx\") pod \"kuadrant-operator-catalog-tf7wp\" (UID: \"f7f87602-3ecb-463c-9956-2af87f8fffb0\") " pod="kuadrant-system/kuadrant-operator-catalog-tf7wp"
Jul 09 23:34:06.528633 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:06.528596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tf7wp"
Jul 09 23:34:06.660727 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:06.660699    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tf7wp"]
Jul 09 23:34:06.662958 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:34:06.662934    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f87602_3ecb_463c_9956_2af87f8fffb0.slice/crio-e0ed040f5759d425fabe21c23b97d3c168427fa9462cbc43be028e90d8d7bdb8 WatchSource:0}: Error finding container e0ed040f5759d425fabe21c23b97d3c168427fa9462cbc43be028e90d8d7bdb8: Status 404 returned error can't find the container with id e0ed040f5759d425fabe21c23b97d3c168427fa9462cbc43be028e90d8d7bdb8
Jul 09 23:34:06.950342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:06.950251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tf7wp" event={"ID":"f7f87602-3ecb-463c-9956-2af87f8fffb0","Type":"ContainerStarted","Data":"e0ed040f5759d425fabe21c23b97d3c168427fa9462cbc43be028e90d8d7bdb8"}
Jul 09 23:34:06.951494 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:06.951464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d2wcm" event={"ID":"08ac9627-256d-4abf-8247-4825a3a4cf5f","Type":"ContainerStarted","Data":"4c05fd217e5ac73effbfd9c09e92a3b8a37eba29d83e9b2e86a0df0ca5c29387"}
Jul 09 23:34:08.962427 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:08.962383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d2wcm" event={"ID":"08ac9627-256d-4abf-8247-4825a3a4cf5f","Type":"ContainerStarted","Data":"37a8e15698dbf3cc43bef8a7b2bb7509a8417475f8c2c4e33753679bf37d3d9a"}
Jul 09 23:34:08.962427 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:08.962414    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-d2wcm" podUID="08ac9627-256d-4abf-8247-4825a3a4cf5f" containerName="registry-server" containerID="cri-o://37a8e15698dbf3cc43bef8a7b2bb7509a8417475f8c2c4e33753679bf37d3d9a" gracePeriod=2
Jul 09 23:34:08.963983 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:08.963954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tf7wp" event={"ID":"f7f87602-3ecb-463c-9956-2af87f8fffb0","Type":"ContainerStarted","Data":"cff307197edadac840cd264360650115401d7f33e92fab92a129a49929e95a97"}
Jul 09 23:34:08.977979 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:08.977932    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-d2wcm" podStartSLOduration=1.7987550369999998 podStartE2EDuration="3.977916715s" totalImagesPullingTime="2.179161678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:34:08.975632834 +0000 UTC m=+367.119563621" watchObservedRunningTime="2026-07-09 23:34:08.977916715 +0000 UTC m=+367.121847501"
Jul 09 23:34:08.992431 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:08.992385    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tf7wp" podStartSLOduration=1.362432466 podStartE2EDuration="2.992372295s" totalImagesPullingTime="1.629939829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:34:08.991011586 +0000 UTC m=+367.134942581" watchObservedRunningTime="2026-07-09 23:34:08.992372295 +0000 UTC m=+367.136303081"
Jul 09 23:34:09.212591 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:09.212532    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d2wcm"
Jul 09 23:34:09.313315 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:09.313276    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wq98\" (UniqueName: \"kubernetes.io/projected/08ac9627-256d-4abf-8247-4825a3a4cf5f-kube-api-access-9wq98\") pod \"08ac9627-256d-4abf-8247-4825a3a4cf5f\" (UID: \"08ac9627-256d-4abf-8247-4825a3a4cf5f\") "
Jul 09 23:34:09.315415 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:09.315392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ac9627-256d-4abf-8247-4825a3a4cf5f-kube-api-access-9wq98" (OuterVolumeSpecName: "kube-api-access-9wq98") pod "08ac9627-256d-4abf-8247-4825a3a4cf5f" (UID: "08ac9627-256d-4abf-8247-4825a3a4cf5f"). InnerVolumeSpecName "kube-api-access-9wq98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:34:09.413751 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:09.413710    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wq98\" (UniqueName: \"kubernetes.io/projected/08ac9627-256d-4abf-8247-4825a3a4cf5f-kube-api-access-9wq98\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:09.969207 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:09.969171    2580 generic.go:356] "Generic (PLEG): container finished" podID="08ac9627-256d-4abf-8247-4825a3a4cf5f" containerID="37a8e15698dbf3cc43bef8a7b2bb7509a8417475f8c2c4e33753679bf37d3d9a" exitCode=0
Jul 09 23:34:09.969667 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:09.969213    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d2wcm"
Jul 09 23:34:09.969667 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:09.969259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d2wcm" event={"ID":"08ac9627-256d-4abf-8247-4825a3a4cf5f","Type":"ContainerDied","Data":"37a8e15698dbf3cc43bef8a7b2bb7509a8417475f8c2c4e33753679bf37d3d9a"}
Jul 09 23:34:09.969667 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:09.969315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d2wcm" event={"ID":"08ac9627-256d-4abf-8247-4825a3a4cf5f","Type":"ContainerDied","Data":"4c05fd217e5ac73effbfd9c09e92a3b8a37eba29d83e9b2e86a0df0ca5c29387"}
Jul 09 23:34:09.969667 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:09.969345    2580 scope.go:117] "RemoveContainer" containerID="37a8e15698dbf3cc43bef8a7b2bb7509a8417475f8c2c4e33753679bf37d3d9a"
Jul 09 23:34:09.979022 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:09.979007    2580 scope.go:117] "RemoveContainer" containerID="37a8e15698dbf3cc43bef8a7b2bb7509a8417475f8c2c4e33753679bf37d3d9a"
Jul 09 23:34:09.979296 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:09.979281    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a8e15698dbf3cc43bef8a7b2bb7509a8417475f8c2c4e33753679bf37d3d9a\": container with ID starting with 37a8e15698dbf3cc43bef8a7b2bb7509a8417475f8c2c4e33753679bf37d3d9a not found: ID does not exist" containerID="37a8e15698dbf3cc43bef8a7b2bb7509a8417475f8c2c4e33753679bf37d3d9a"
Jul 09 23:34:09.979339 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:09.979304    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a8e15698dbf3cc43bef8a7b2bb7509a8417475f8c2c4e33753679bf37d3d9a"} err="failed to get container status \"37a8e15698dbf3cc43bef8a7b2bb7509a8417475f8c2c4e33753679bf37d3d9a\": rpc error: code = NotFound desc = could not find container \"37a8e15698dbf3cc43bef8a7b2bb7509a8417475f8c2c4e33753679bf37d3d9a\": container with ID starting with 37a8e15698dbf3cc43bef8a7b2bb7509a8417475f8c2c4e33753679bf37d3d9a not found: ID does not exist"
Jul 09 23:34:09.988357 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:09.988316    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d2wcm"]
Jul 09 23:34:09.990224 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:09.990206    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d2wcm"]
Jul 09 23:34:10.418790 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:10.418759    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08ac9627-256d-4abf-8247-4825a3a4cf5f" path="/var/lib/kubelet/pods/08ac9627-256d-4abf-8247-4825a3a4cf5f/volumes"
Jul 09 23:34:16.528823 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:16.528778    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-tf7wp"
Jul 09 23:34:16.529393 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:16.528853    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-tf7wp"
Jul 09 23:34:16.551185 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:16.551153    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-tf7wp"
Jul 09 23:34:17.025102 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:17.025071    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-tf7wp"
Jul 09 23:34:21.447856 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.447821    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"]
Jul 09 23:34:21.448249 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.448114    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ac9627-256d-4abf-8247-4825a3a4cf5f" containerName="registry-server"
Jul 09 23:34:21.448249 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.448125    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ac9627-256d-4abf-8247-4825a3a4cf5f" containerName="registry-server"
Jul 09 23:34:21.448249 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.448192    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ac9627-256d-4abf-8247-4825a3a4cf5f" containerName="registry-server"
Jul 09 23:34:21.452853 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.452832    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"
Jul 09 23:34:21.455137 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.455119    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xk9b9\""
Jul 09 23:34:21.458274 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.458245    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"]
Jul 09 23:34:21.614792 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.614754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw\" (UID: \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"
Jul 09 23:34:21.614982 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.614820    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxzhw\" (UniqueName: \"kubernetes.io/projected/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-kube-api-access-kxzhw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw\" (UID: \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"
Jul 09 23:34:21.614982 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.614885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw\" (UID: \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"
Jul 09 23:34:21.715912 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.715808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw\" (UID: \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"
Jul 09 23:34:21.715912 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.715879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxzhw\" (UniqueName: \"kubernetes.io/projected/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-kube-api-access-kxzhw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw\" (UID: \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"
Jul 09 23:34:21.715912 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.715911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw\" (UID: \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"
Jul 09 23:34:21.716215 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.716191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw\" (UID: \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"
Jul 09 23:34:21.716289 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.716269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw\" (UID: \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"
Jul 09 23:34:21.723723 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.723690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxzhw\" (UniqueName: \"kubernetes.io/projected/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-kube-api-access-kxzhw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw\" (UID: \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"
Jul 09 23:34:21.763388 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.763352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"
Jul 09 23:34:21.891577 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:21.891446    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"]
Jul 09 23:34:21.894339 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:34:21.894314    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ad7126_c53c_4cc6_bcde_dc0692fe9fe4.slice/crio-da364656a5f83df620687380ed200b67d432d820c97c2becbe92fdf4a56710da WatchSource:0}: Error finding container da364656a5f83df620687380ed200b67d432d820c97c2becbe92fdf4a56710da: Status 404 returned error can't find the container with id da364656a5f83df620687380ed200b67d432d820c97c2becbe92fdf4a56710da
Jul 09 23:34:22.025445 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.025409    2580 generic.go:356] "Generic (PLEG): container finished" podID="24ad7126-c53c-4cc6-bcde-dc0692fe9fe4" containerID="89fdfdb5c92cd0c4a8895afd77a5c2bfc07fbdd225adde417d1ed7c0a6fc64dd" exitCode=0
Jul 09 23:34:22.025445 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.025448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw" event={"ID":"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4","Type":"ContainerDied","Data":"89fdfdb5c92cd0c4a8895afd77a5c2bfc07fbdd225adde417d1ed7c0a6fc64dd"}
Jul 09 23:34:22.025804 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.025485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw" event={"ID":"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4","Type":"ContainerStarted","Data":"da364656a5f83df620687380ed200b67d432d820c97c2becbe92fdf4a56710da"}
Jul 09 23:34:22.244919 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.244835    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"]
Jul 09 23:34:22.255272 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.255217    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"
Jul 09 23:34:22.256724 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.256699    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"]
Jul 09 23:34:22.422440 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.422411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/710880cd-b71d-4525-8dc9-d51ba1b001fe-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9\" (UID: \"710880cd-b71d-4525-8dc9-d51ba1b001fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"
Jul 09 23:34:22.422619 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.422464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/710880cd-b71d-4525-8dc9-d51ba1b001fe-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9\" (UID: \"710880cd-b71d-4525-8dc9-d51ba1b001fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"
Jul 09 23:34:22.422619 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.422498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcrg6\" (UniqueName: \"kubernetes.io/projected/710880cd-b71d-4525-8dc9-d51ba1b001fe-kube-api-access-qcrg6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9\" (UID: \"710880cd-b71d-4525-8dc9-d51ba1b001fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"
Jul 09 23:34:22.522941 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.522902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/710880cd-b71d-4525-8dc9-d51ba1b001fe-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9\" (UID: \"710880cd-b71d-4525-8dc9-d51ba1b001fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"
Jul 09 23:34:22.523314 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.522954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/710880cd-b71d-4525-8dc9-d51ba1b001fe-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9\" (UID: \"710880cd-b71d-4525-8dc9-d51ba1b001fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"
Jul 09 23:34:22.523314 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.522989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcrg6\" (UniqueName: \"kubernetes.io/projected/710880cd-b71d-4525-8dc9-d51ba1b001fe-kube-api-access-qcrg6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9\" (UID: \"710880cd-b71d-4525-8dc9-d51ba1b001fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"
Jul 09 23:34:22.523314 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.523293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/710880cd-b71d-4525-8dc9-d51ba1b001fe-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9\" (UID: \"710880cd-b71d-4525-8dc9-d51ba1b001fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"
Jul 09 23:34:22.523455 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.523347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/710880cd-b71d-4525-8dc9-d51ba1b001fe-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9\" (UID: \"710880cd-b71d-4525-8dc9-d51ba1b001fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"
Jul 09 23:34:22.531151 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.531119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcrg6\" (UniqueName: \"kubernetes.io/projected/710880cd-b71d-4525-8dc9-d51ba1b001fe-kube-api-access-qcrg6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9\" (UID: \"710880cd-b71d-4525-8dc9-d51ba1b001fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"
Jul 09 23:34:22.565380 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.565340    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"
Jul 09 23:34:22.647533 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.647502    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"]
Jul 09 23:34:22.651941 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.651917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"
Jul 09 23:34:22.658292 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.658247    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"]
Jul 09 23:34:22.695809 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.695782    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"]
Jul 09 23:34:22.697552 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:34:22.697524    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710880cd_b71d_4525_8dc9_d51ba1b001fe.slice/crio-d2d50374d7d4fd7800ad7abc1b8bdd8a4974955de4706df3ce154189f40a15f8 WatchSource:0}: Error finding container d2d50374d7d4fd7800ad7abc1b8bdd8a4974955de4706df3ce154189f40a15f8: Status 404 returned error can't find the container with id d2d50374d7d4fd7800ad7abc1b8bdd8a4974955de4706df3ce154189f40a15f8
Jul 09 23:34:22.725669 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.725631    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b501a26c-47ca-4d86-9d34-37c70645e5c5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw\" (UID: \"b501a26c-47ca-4d86-9d34-37c70645e5c5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"
Jul 09 23:34:22.725780 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.725734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b501a26c-47ca-4d86-9d34-37c70645e5c5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw\" (UID: \"b501a26c-47ca-4d86-9d34-37c70645e5c5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"
Jul 09 23:34:22.725780 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.725765    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljcwf\" (UniqueName: \"kubernetes.io/projected/b501a26c-47ca-4d86-9d34-37c70645e5c5-kube-api-access-ljcwf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw\" (UID: \"b501a26c-47ca-4d86-9d34-37c70645e5c5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"
Jul 09 23:34:22.826683 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.826633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b501a26c-47ca-4d86-9d34-37c70645e5c5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw\" (UID: \"b501a26c-47ca-4d86-9d34-37c70645e5c5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"
Jul 09 23:34:22.826683 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.826683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljcwf\" (UniqueName: \"kubernetes.io/projected/b501a26c-47ca-4d86-9d34-37c70645e5c5-kube-api-access-ljcwf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw\" (UID: \"b501a26c-47ca-4d86-9d34-37c70645e5c5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"
Jul 09 23:34:22.826875 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.826737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b501a26c-47ca-4d86-9d34-37c70645e5c5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw\" (UID: \"b501a26c-47ca-4d86-9d34-37c70645e5c5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"
Jul 09 23:34:22.827112 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.827081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b501a26c-47ca-4d86-9d34-37c70645e5c5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw\" (UID: \"b501a26c-47ca-4d86-9d34-37c70645e5c5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"
Jul 09 23:34:22.827112 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.827098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b501a26c-47ca-4d86-9d34-37c70645e5c5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw\" (UID: \"b501a26c-47ca-4d86-9d34-37c70645e5c5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"
Jul 09 23:34:22.834302 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.834278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljcwf\" (UniqueName: \"kubernetes.io/projected/b501a26c-47ca-4d86-9d34-37c70645e5c5-kube-api-access-ljcwf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw\" (UID: \"b501a26c-47ca-4d86-9d34-37c70645e5c5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"
Jul 09 23:34:22.963473 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:22.963449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"
Jul 09 23:34:23.032677 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.032626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw" event={"ID":"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4","Type":"ContainerStarted","Data":"b0dcade0f028212820119f4eab8b5e5dda1fc89229c28e279d21038bad065515"}
Jul 09 23:34:23.034188 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.034162    2580 generic.go:356] "Generic (PLEG): container finished" podID="710880cd-b71d-4525-8dc9-d51ba1b001fe" containerID="c4baa408e690b00b37a12c634f48b6082bc3a2a926d2c6d2205be2fb4cdeba99" exitCode=0
Jul 09 23:34:23.034306 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.034225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9" event={"ID":"710880cd-b71d-4525-8dc9-d51ba1b001fe","Type":"ContainerDied","Data":"c4baa408e690b00b37a12c634f48b6082bc3a2a926d2c6d2205be2fb4cdeba99"}
Jul 09 23:34:23.034306 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.034280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9" event={"ID":"710880cd-b71d-4525-8dc9-d51ba1b001fe","Type":"ContainerStarted","Data":"d2d50374d7d4fd7800ad7abc1b8bdd8a4974955de4706df3ce154189f40a15f8"}
Jul 09 23:34:23.047093 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.047064    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"]
Jul 09 23:34:23.054570 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.053905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"
Jul 09 23:34:23.059205 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.059071    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"]
Jul 09 23:34:23.105182 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.105153    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"]
Jul 09 23:34:23.108696 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:34:23.108669    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb501a26c_47ca_4d86_9d34_37c70645e5c5.slice/crio-77234611054bcb5652bbc7b9e8221efa11015d93cd113e867edae44ddc608c92 WatchSource:0}: Error finding container 77234611054bcb5652bbc7b9e8221efa11015d93cd113e867edae44ddc608c92: Status 404 returned error can't find the container with id 77234611054bcb5652bbc7b9e8221efa11015d93cd113e867edae44ddc608c92
Jul 09 23:34:23.128848 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.128821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95bf198f-c90a-400e-ac71-52ec87b341c3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm\" (UID: \"95bf198f-c90a-400e-ac71-52ec87b341c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"
Jul 09 23:34:23.128963 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.128858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdd8q\" (UniqueName: \"kubernetes.io/projected/95bf198f-c90a-400e-ac71-52ec87b341c3-kube-api-access-kdd8q\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm\" (UID: \"95bf198f-c90a-400e-ac71-52ec87b341c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"
Jul 09 23:34:23.129028 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.128995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95bf198f-c90a-400e-ac71-52ec87b341c3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm\" (UID: \"95bf198f-c90a-400e-ac71-52ec87b341c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"
Jul 09 23:34:23.229939 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.229903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95bf198f-c90a-400e-ac71-52ec87b341c3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm\" (UID: \"95bf198f-c90a-400e-ac71-52ec87b341c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"
Jul 09 23:34:23.229939 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.229938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdd8q\" (UniqueName: \"kubernetes.io/projected/95bf198f-c90a-400e-ac71-52ec87b341c3-kube-api-access-kdd8q\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm\" (UID: \"95bf198f-c90a-400e-ac71-52ec87b341c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"
Jul 09 23:34:23.230128 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.229995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95bf198f-c90a-400e-ac71-52ec87b341c3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm\" (UID: \"95bf198f-c90a-400e-ac71-52ec87b341c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"
Jul 09 23:34:23.230297 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.230277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95bf198f-c90a-400e-ac71-52ec87b341c3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm\" (UID: \"95bf198f-c90a-400e-ac71-52ec87b341c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"
Jul 09 23:34:23.230352 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.230306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95bf198f-c90a-400e-ac71-52ec87b341c3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm\" (UID: \"95bf198f-c90a-400e-ac71-52ec87b341c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"
Jul 09 23:34:23.238444 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.238417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdd8q\" (UniqueName: \"kubernetes.io/projected/95bf198f-c90a-400e-ac71-52ec87b341c3-kube-api-access-kdd8q\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm\" (UID: \"95bf198f-c90a-400e-ac71-52ec87b341c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"
Jul 09 23:34:23.411343 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.411237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"
Jul 09 23:34:23.540484 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:23.540449    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"]
Jul 09 23:34:23.542632 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:34:23.542595    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95bf198f_c90a_400e_ac71_52ec87b341c3.slice/crio-83c8f3b2dc8ffa047fc0b62e833c890cd2b874766875ca60004d061af28d7b2c WatchSource:0}: Error finding container 83c8f3b2dc8ffa047fc0b62e833c890cd2b874766875ca60004d061af28d7b2c: Status 404 returned error can't find the container with id 83c8f3b2dc8ffa047fc0b62e833c890cd2b874766875ca60004d061af28d7b2c
Jul 09 23:34:24.043721 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:24.043684    2580 generic.go:356] "Generic (PLEG): container finished" podID="b501a26c-47ca-4d86-9d34-37c70645e5c5" containerID="f6961bb0871f91b80451d982482ab689cd706a2031fd12ddd00f9d77f54509ef" exitCode=0
Jul 09 23:34:24.043897 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:24.043832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw" event={"ID":"b501a26c-47ca-4d86-9d34-37c70645e5c5","Type":"ContainerDied","Data":"f6961bb0871f91b80451d982482ab689cd706a2031fd12ddd00f9d77f54509ef"}
Jul 09 23:34:24.043897 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:24.043886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw" event={"ID":"b501a26c-47ca-4d86-9d34-37c70645e5c5","Type":"ContainerStarted","Data":"77234611054bcb5652bbc7b9e8221efa11015d93cd113e867edae44ddc608c92"}
Jul 09 23:34:24.045796 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:24.045775    2580 generic.go:356] "Generic (PLEG): container finished" podID="24ad7126-c53c-4cc6-bcde-dc0692fe9fe4" containerID="b0dcade0f028212820119f4eab8b5e5dda1fc89229c28e279d21038bad065515" exitCode=0
Jul 09 23:34:24.045897 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:24.045840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw" event={"ID":"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4","Type":"ContainerDied","Data":"b0dcade0f028212820119f4eab8b5e5dda1fc89229c28e279d21038bad065515"}
Jul 09 23:34:24.047838 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:24.047814    2580 generic.go:356] "Generic (PLEG): container finished" podID="710880cd-b71d-4525-8dc9-d51ba1b001fe" containerID="8285ad1316e490cc81cf64013bd382b4f6f3c11555a34172eefdc26a2162a965" exitCode=0
Jul 09 23:34:24.048115 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:24.047900    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9" event={"ID":"710880cd-b71d-4525-8dc9-d51ba1b001fe","Type":"ContainerDied","Data":"8285ad1316e490cc81cf64013bd382b4f6f3c11555a34172eefdc26a2162a965"}
Jul 09 23:34:24.049471 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:24.049453    2580 generic.go:356] "Generic (PLEG): container finished" podID="95bf198f-c90a-400e-ac71-52ec87b341c3" containerID="001646257df272f7067de0e8527f18c7c1d8c42a064a764c607f9b32018d5e87" exitCode=0
Jul 09 23:34:24.049530 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:24.049518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm" event={"ID":"95bf198f-c90a-400e-ac71-52ec87b341c3","Type":"ContainerDied","Data":"001646257df272f7067de0e8527f18c7c1d8c42a064a764c607f9b32018d5e87"}
Jul 09 23:34:24.049597 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:24.049557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm" event={"ID":"95bf198f-c90a-400e-ac71-52ec87b341c3","Type":"ContainerStarted","Data":"83c8f3b2dc8ffa047fc0b62e833c890cd2b874766875ca60004d061af28d7b2c"}
Jul 09 23:34:25.055401 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:25.055365    2580 generic.go:356] "Generic (PLEG): container finished" podID="b501a26c-47ca-4d86-9d34-37c70645e5c5" containerID="db9234bd61f7578701b64c84266d7baf6595d2fd0575d28af749aaa9ae0e9cff" exitCode=0
Jul 09 23:34:25.055863 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:25.055453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw" event={"ID":"b501a26c-47ca-4d86-9d34-37c70645e5c5","Type":"ContainerDied","Data":"db9234bd61f7578701b64c84266d7baf6595d2fd0575d28af749aaa9ae0e9cff"}
Jul 09 23:34:25.057703 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:25.057640    2580 generic.go:356] "Generic (PLEG): container finished" podID="24ad7126-c53c-4cc6-bcde-dc0692fe9fe4" containerID="d818aed50719ed4267cb32f148af1c3d5c5ac4b36bcc08bc2336ff0316c80481" exitCode=0
Jul 09 23:34:25.057703 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:25.057681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw" event={"ID":"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4","Type":"ContainerDied","Data":"d818aed50719ed4267cb32f148af1c3d5c5ac4b36bcc08bc2336ff0316c80481"}
Jul 09 23:34:25.059701 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:25.059682    2580 generic.go:356] "Generic (PLEG): container finished" podID="710880cd-b71d-4525-8dc9-d51ba1b001fe" containerID="961d55a153d51f323459fd2fb59937660c58e4e84f33d11ac25a564c31ae099f" exitCode=0
Jul 09 23:34:25.059800 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:25.059755    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9" event={"ID":"710880cd-b71d-4525-8dc9-d51ba1b001fe","Type":"ContainerDied","Data":"961d55a153d51f323459fd2fb59937660c58e4e84f33d11ac25a564c31ae099f"}
Jul 09 23:34:25.061484 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:25.061455    2580 generic.go:356] "Generic (PLEG): container finished" podID="95bf198f-c90a-400e-ac71-52ec87b341c3" containerID="3d661ca5446741d9e4e9d33b50d538ef4f90ee7533f4ac8c863571dc752be6e7" exitCode=0
Jul 09 23:34:25.061577 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:25.061480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm" event={"ID":"95bf198f-c90a-400e-ac71-52ec87b341c3","Type":"ContainerDied","Data":"3d661ca5446741d9e4e9d33b50d538ef4f90ee7533f4ac8c863571dc752be6e7"}
Jul 09 23:34:26.068985 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.068947    2580 generic.go:356] "Generic (PLEG): container finished" podID="95bf198f-c90a-400e-ac71-52ec87b341c3" containerID="6ed62187a01f684b522929bef29b8522b7c54368222127d2416b2f8e9a8bced0" exitCode=0
Jul 09 23:34:26.069372 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.069040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm" event={"ID":"95bf198f-c90a-400e-ac71-52ec87b341c3","Type":"ContainerDied","Data":"6ed62187a01f684b522929bef29b8522b7c54368222127d2416b2f8e9a8bced0"}
Jul 09 23:34:26.071049 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.071027    2580 generic.go:356] "Generic (PLEG): container finished" podID="b501a26c-47ca-4d86-9d34-37c70645e5c5" containerID="4f9b94e51b6875cce80056b5e206fd1d0d9c5b6e850fe2fb98b944f9656a0045" exitCode=0
Jul 09 23:34:26.076514 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.071534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw" event={"ID":"b501a26c-47ca-4d86-9d34-37c70645e5c5","Type":"ContainerDied","Data":"4f9b94e51b6875cce80056b5e206fd1d0d9c5b6e850fe2fb98b944f9656a0045"}
Jul 09 23:34:26.230996 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.230972    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"
Jul 09 23:34:26.233827 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.233806    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"
Jul 09 23:34:26.357786 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.357676    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-util\") pod \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\" (UID: \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\") "
Jul 09 23:34:26.357786 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.357758    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-bundle\") pod \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\" (UID: \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\") "
Jul 09 23:34:26.358016 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.357797    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/710880cd-b71d-4525-8dc9-d51ba1b001fe-util\") pod \"710880cd-b71d-4525-8dc9-d51ba1b001fe\" (UID: \"710880cd-b71d-4525-8dc9-d51ba1b001fe\") "
Jul 09 23:34:26.358016 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.357835    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcrg6\" (UniqueName: \"kubernetes.io/projected/710880cd-b71d-4525-8dc9-d51ba1b001fe-kube-api-access-qcrg6\") pod \"710880cd-b71d-4525-8dc9-d51ba1b001fe\" (UID: \"710880cd-b71d-4525-8dc9-d51ba1b001fe\") "
Jul 09 23:34:26.358016 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.357861    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxzhw\" (UniqueName: \"kubernetes.io/projected/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-kube-api-access-kxzhw\") pod \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\" (UID: \"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4\") "
Jul 09 23:34:26.358016 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.357917    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/710880cd-b71d-4525-8dc9-d51ba1b001fe-bundle\") pod \"710880cd-b71d-4525-8dc9-d51ba1b001fe\" (UID: \"710880cd-b71d-4525-8dc9-d51ba1b001fe\") "
Jul 09 23:34:26.358395 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.358316    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-bundle" (OuterVolumeSpecName: "bundle") pod "24ad7126-c53c-4cc6-bcde-dc0692fe9fe4" (UID: "24ad7126-c53c-4cc6-bcde-dc0692fe9fe4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:34:26.358554 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.358528    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710880cd-b71d-4525-8dc9-d51ba1b001fe-bundle" (OuterVolumeSpecName: "bundle") pod "710880cd-b71d-4525-8dc9-d51ba1b001fe" (UID: "710880cd-b71d-4525-8dc9-d51ba1b001fe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:34:26.360089 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.360057    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710880cd-b71d-4525-8dc9-d51ba1b001fe-kube-api-access-qcrg6" (OuterVolumeSpecName: "kube-api-access-qcrg6") pod "710880cd-b71d-4525-8dc9-d51ba1b001fe" (UID: "710880cd-b71d-4525-8dc9-d51ba1b001fe"). InnerVolumeSpecName "kube-api-access-qcrg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:34:26.360499 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.360478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-kube-api-access-kxzhw" (OuterVolumeSpecName: "kube-api-access-kxzhw") pod "24ad7126-c53c-4cc6-bcde-dc0692fe9fe4" (UID: "24ad7126-c53c-4cc6-bcde-dc0692fe9fe4"). InnerVolumeSpecName "kube-api-access-kxzhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:34:26.363898 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.363878    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710880cd-b71d-4525-8dc9-d51ba1b001fe-util" (OuterVolumeSpecName: "util") pod "710880cd-b71d-4525-8dc9-d51ba1b001fe" (UID: "710880cd-b71d-4525-8dc9-d51ba1b001fe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:34:26.364090 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.364071    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-util" (OuterVolumeSpecName: "util") pod "24ad7126-c53c-4cc6-bcde-dc0692fe9fe4" (UID: "24ad7126-c53c-4cc6-bcde-dc0692fe9fe4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:34:26.459030 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.458998    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/710880cd-b71d-4525-8dc9-d51ba1b001fe-bundle\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:26.459030 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.459025    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-util\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:26.459030 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.459033    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-bundle\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:26.459226 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.459041    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/710880cd-b71d-4525-8dc9-d51ba1b001fe-util\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:26.459226 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.459049    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcrg6\" (UniqueName: \"kubernetes.io/projected/710880cd-b71d-4525-8dc9-d51ba1b001fe-kube-api-access-qcrg6\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:26.459226 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:26.459059    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxzhw\" (UniqueName: \"kubernetes.io/projected/24ad7126-c53c-4cc6-bcde-dc0692fe9fe4-kube-api-access-kxzhw\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:27.077416 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.077373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw" event={"ID":"24ad7126-c53c-4cc6-bcde-dc0692fe9fe4","Type":"ContainerDied","Data":"da364656a5f83df620687380ed200b67d432d820c97c2becbe92fdf4a56710da"}
Jul 09 23:34:27.077942 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.077423    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da364656a5f83df620687380ed200b67d432d820c97c2becbe92fdf4a56710da"
Jul 09 23:34:27.077942 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.077384    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw"
Jul 09 23:34:27.079542 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.079499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9" event={"ID":"710880cd-b71d-4525-8dc9-d51ba1b001fe","Type":"ContainerDied","Data":"d2d50374d7d4fd7800ad7abc1b8bdd8a4974955de4706df3ce154189f40a15f8"}
Jul 09 23:34:27.079674 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.079562    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2d50374d7d4fd7800ad7abc1b8bdd8a4974955de4706df3ce154189f40a15f8"
Jul 09 23:34:27.079674 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.079526    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9"
Jul 09 23:34:27.238552 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.238529    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"
Jul 09 23:34:27.240664 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.240638    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"
Jul 09 23:34:27.365070 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.364983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljcwf\" (UniqueName: \"kubernetes.io/projected/b501a26c-47ca-4d86-9d34-37c70645e5c5-kube-api-access-ljcwf\") pod \"b501a26c-47ca-4d86-9d34-37c70645e5c5\" (UID: \"b501a26c-47ca-4d86-9d34-37c70645e5c5\") "
Jul 09 23:34:27.365070 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.365052    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b501a26c-47ca-4d86-9d34-37c70645e5c5-bundle\") pod \"b501a26c-47ca-4d86-9d34-37c70645e5c5\" (UID: \"b501a26c-47ca-4d86-9d34-37c70645e5c5\") "
Jul 09 23:34:27.365070 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.365079    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95bf198f-c90a-400e-ac71-52ec87b341c3-util\") pod \"95bf198f-c90a-400e-ac71-52ec87b341c3\" (UID: \"95bf198f-c90a-400e-ac71-52ec87b341c3\") "
Jul 09 23:34:27.365343 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.365094    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdd8q\" (UniqueName: \"kubernetes.io/projected/95bf198f-c90a-400e-ac71-52ec87b341c3-kube-api-access-kdd8q\") pod \"95bf198f-c90a-400e-ac71-52ec87b341c3\" (UID: \"95bf198f-c90a-400e-ac71-52ec87b341c3\") "
Jul 09 23:34:27.365343 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.365116    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b501a26c-47ca-4d86-9d34-37c70645e5c5-util\") pod \"b501a26c-47ca-4d86-9d34-37c70645e5c5\" (UID: \"b501a26c-47ca-4d86-9d34-37c70645e5c5\") "
Jul 09 23:34:27.365343 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.365134    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95bf198f-c90a-400e-ac71-52ec87b341c3-bundle\") pod \"95bf198f-c90a-400e-ac71-52ec87b341c3\" (UID: \"95bf198f-c90a-400e-ac71-52ec87b341c3\") "
Jul 09 23:34:27.365796 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.365759    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b501a26c-47ca-4d86-9d34-37c70645e5c5-bundle" (OuterVolumeSpecName: "bundle") pod "b501a26c-47ca-4d86-9d34-37c70645e5c5" (UID: "b501a26c-47ca-4d86-9d34-37c70645e5c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:34:27.365902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.365767    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bf198f-c90a-400e-ac71-52ec87b341c3-bundle" (OuterVolumeSpecName: "bundle") pod "95bf198f-c90a-400e-ac71-52ec87b341c3" (UID: "95bf198f-c90a-400e-ac71-52ec87b341c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:34:27.367303 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.367275    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b501a26c-47ca-4d86-9d34-37c70645e5c5-kube-api-access-ljcwf" (OuterVolumeSpecName: "kube-api-access-ljcwf") pod "b501a26c-47ca-4d86-9d34-37c70645e5c5" (UID: "b501a26c-47ca-4d86-9d34-37c70645e5c5"). InnerVolumeSpecName "kube-api-access-ljcwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:34:27.367430 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.367406    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95bf198f-c90a-400e-ac71-52ec87b341c3-kube-api-access-kdd8q" (OuterVolumeSpecName: "kube-api-access-kdd8q") pod "95bf198f-c90a-400e-ac71-52ec87b341c3" (UID: "95bf198f-c90a-400e-ac71-52ec87b341c3"). InnerVolumeSpecName "kube-api-access-kdd8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:34:27.373780 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.373753    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bf198f-c90a-400e-ac71-52ec87b341c3-util" (OuterVolumeSpecName: "util") pod "95bf198f-c90a-400e-ac71-52ec87b341c3" (UID: "95bf198f-c90a-400e-ac71-52ec87b341c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:34:27.374362 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.374345    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b501a26c-47ca-4d86-9d34-37c70645e5c5-util" (OuterVolumeSpecName: "util") pod "b501a26c-47ca-4d86-9d34-37c70645e5c5" (UID: "b501a26c-47ca-4d86-9d34-37c70645e5c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:34:27.466000 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.465952    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b501a26c-47ca-4d86-9d34-37c70645e5c5-bundle\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:27.466000 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.465994    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95bf198f-c90a-400e-ac71-52ec87b341c3-util\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:27.466000 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.466004    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdd8q\" (UniqueName: \"kubernetes.io/projected/95bf198f-c90a-400e-ac71-52ec87b341c3-kube-api-access-kdd8q\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:27.466000 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.466013    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b501a26c-47ca-4d86-9d34-37c70645e5c5-util\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:27.466253 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.466024    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95bf198f-c90a-400e-ac71-52ec87b341c3-bundle\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:27.466253 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:27.466032    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljcwf\" (UniqueName: \"kubernetes.io/projected/b501a26c-47ca-4d86-9d34-37c70645e5c5-kube-api-access-ljcwf\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:28.086267 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:28.086238    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm"
Jul 09 23:34:28.086267 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:28.086252    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm" event={"ID":"95bf198f-c90a-400e-ac71-52ec87b341c3","Type":"ContainerDied","Data":"83c8f3b2dc8ffa047fc0b62e833c890cd2b874766875ca60004d061af28d7b2c"}
Jul 09 23:34:28.086825 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:28.086300    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83c8f3b2dc8ffa047fc0b62e833c890cd2b874766875ca60004d061af28d7b2c"
Jul 09 23:34:28.088198 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:28.088166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw" event={"ID":"b501a26c-47ca-4d86-9d34-37c70645e5c5","Type":"ContainerDied","Data":"77234611054bcb5652bbc7b9e8221efa11015d93cd113e867edae44ddc608c92"}
Jul 09 23:34:28.088311 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:28.088209    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77234611054bcb5652bbc7b9e8221efa11015d93cd113e867edae44ddc608c92"
Jul 09 23:34:28.088311 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:28.088174    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw"
Jul 09 23:34:30.530682 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.529834    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"]
Jul 09 23:34:30.530682 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530543    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ad7126-c53c-4cc6-bcde-dc0692fe9fe4" containerName="pull"
Jul 09 23:34:30.530682 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530562    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ad7126-c53c-4cc6-bcde-dc0692fe9fe4" containerName="pull"
Jul 09 23:34:30.530682 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530587    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ad7126-c53c-4cc6-bcde-dc0692fe9fe4" containerName="util"
Jul 09 23:34:30.530682 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530595    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ad7126-c53c-4cc6-bcde-dc0692fe9fe4" containerName="util"
Jul 09 23:34:30.530682 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530613    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95bf198f-c90a-400e-ac71-52ec87b341c3" containerName="extract"
Jul 09 23:34:30.530682 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530621    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="95bf198f-c90a-400e-ac71-52ec87b341c3" containerName="extract"
Jul 09 23:34:30.530682 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530640    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95bf198f-c90a-400e-ac71-52ec87b341c3" containerName="util"
Jul 09 23:34:30.530682 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530648    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="95bf198f-c90a-400e-ac71-52ec87b341c3" containerName="util"
Jul 09 23:34:30.530682 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530683    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b501a26c-47ca-4d86-9d34-37c70645e5c5" containerName="extract"
Jul 09 23:34:30.530682 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530691    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b501a26c-47ca-4d86-9d34-37c70645e5c5" containerName="extract"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530709    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b501a26c-47ca-4d86-9d34-37c70645e5c5" containerName="pull"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530717    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b501a26c-47ca-4d86-9d34-37c70645e5c5" containerName="pull"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530728    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ad7126-c53c-4cc6-bcde-dc0692fe9fe4" containerName="extract"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530735    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ad7126-c53c-4cc6-bcde-dc0692fe9fe4" containerName="extract"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530756    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95bf198f-c90a-400e-ac71-52ec87b341c3" containerName="pull"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530764    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="95bf198f-c90a-400e-ac71-52ec87b341c3" containerName="pull"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530773    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="710880cd-b71d-4525-8dc9-d51ba1b001fe" containerName="pull"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530780    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="710880cd-b71d-4525-8dc9-d51ba1b001fe" containerName="pull"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530789    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="710880cd-b71d-4525-8dc9-d51ba1b001fe" containerName="util"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530803    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="710880cd-b71d-4525-8dc9-d51ba1b001fe" containerName="util"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530812    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="710880cd-b71d-4525-8dc9-d51ba1b001fe" containerName="extract"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530819    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="710880cd-b71d-4525-8dc9-d51ba1b001fe" containerName="extract"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530839    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b501a26c-47ca-4d86-9d34-37c70645e5c5" containerName="util"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.530846    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b501a26c-47ca-4d86-9d34-37c70645e5c5" containerName="util"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.531021    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ad7126-c53c-4cc6-bcde-dc0692fe9fe4" containerName="extract"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.531031    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b501a26c-47ca-4d86-9d34-37c70645e5c5" containerName="extract"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.531049    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="95bf198f-c90a-400e-ac71-52ec87b341c3" containerName="extract"
Jul 09 23:34:30.531581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.531058    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="710880cd-b71d-4525-8dc9-d51ba1b001fe" containerName="extract"
Jul 09 23:34:30.535961 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.535936    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:30.538616 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.538584    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9blq5\""
Jul 09 23:34:30.541337 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.541310    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"]
Jul 09 23:34:30.696451 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.696413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbx8q\" (UniqueName: \"kubernetes.io/projected/f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645-kube-api-access-pbx8q\") pod \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" (UID: \"f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:30.797059 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.796952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbx8q\" (UniqueName: \"kubernetes.io/projected/f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645-kube-api-access-pbx8q\") pod \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" (UID: \"f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:30.806325 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.806294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbx8q\" (UniqueName: \"kubernetes.io/projected/f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645-kube-api-access-pbx8q\") pod \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" (UID: \"f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:30.847252 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.847209    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:30.976815 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:30.976767    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"]
Jul 09 23:34:30.979164 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:34:30.979141    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b0f3c9_7eb5_4f88_84a6_fc1a19a52645.slice/crio-4f2e0fad60d61f3c23c02e4d5f42d31e7ac63068d0032351e8e1d5934fa8a04d WatchSource:0}: Error finding container 4f2e0fad60d61f3c23c02e4d5f42d31e7ac63068d0032351e8e1d5934fa8a04d: Status 404 returned error can't find the container with id 4f2e0fad60d61f3c23c02e4d5f42d31e7ac63068d0032351e8e1d5934fa8a04d
Jul 09 23:34:31.105635 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:31.105548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp" event={"ID":"f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645","Type":"ContainerStarted","Data":"4f2e0fad60d61f3c23c02e4d5f42d31e7ac63068d0032351e8e1d5934fa8a04d"}
Jul 09 23:34:33.118296 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:33.118260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp" event={"ID":"f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645","Type":"ContainerStarted","Data":"fa8cbc628b5bcf5c9e39c55132ca19fcfbf44866d7126d2cb7992f31be1ff190"}
Jul 09 23:34:33.118696 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:33.118368    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:33.141477 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:33.141425    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp" podStartSLOduration=1.503967251 podStartE2EDuration="3.141410648s" totalImagesPullingTime="1.637443397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:34:33.139763413 +0000 UTC m=+391.283694210" watchObservedRunningTime="2026-07-09 23:34:33.141410648 +0000 UTC m=+391.285341434"
Jul 09 23:34:36.100583 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:36.100549    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"]
Jul 09 23:34:36.103868 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:36.103846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:36.106349 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:36.106327    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9tb6j\""
Jul 09 23:34:36.113513 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:36.113484    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"]
Jul 09 23:34:36.241821 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:36.241779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32d804c5-f92b-4463-a737-eb395fe8af1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" (UID: \"32d804c5-f92b-4463-a737-eb395fe8af1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:36.242022 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:36.241857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dqm8\" (UniqueName: \"kubernetes.io/projected/32d804c5-f92b-4463-a737-eb395fe8af1a-kube-api-access-6dqm8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" (UID: \"32d804c5-f92b-4463-a737-eb395fe8af1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:36.343255 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:36.343214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dqm8\" (UniqueName: \"kubernetes.io/projected/32d804c5-f92b-4463-a737-eb395fe8af1a-kube-api-access-6dqm8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" (UID: \"32d804c5-f92b-4463-a737-eb395fe8af1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:36.343388 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:36.343273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32d804c5-f92b-4463-a737-eb395fe8af1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" (UID: \"32d804c5-f92b-4463-a737-eb395fe8af1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:36.343615 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:36.343596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32d804c5-f92b-4463-a737-eb395fe8af1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" (UID: \"32d804c5-f92b-4463-a737-eb395fe8af1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:36.354811 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:36.354758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dqm8\" (UniqueName: \"kubernetes.io/projected/32d804c5-f92b-4463-a737-eb395fe8af1a-kube-api-access-6dqm8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" (UID: \"32d804c5-f92b-4463-a737-eb395fe8af1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:36.414490 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:36.414455    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:36.561891 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:36.561864    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"]
Jul 09 23:34:36.564601 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:34:36.564576    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d804c5_f92b_4463_a737_eb395fe8af1a.slice/crio-1d730c3cf09eccf014665e96f82e9ce1682be693242ef73a3421ca8f541a4b92 WatchSource:0}: Error finding container 1d730c3cf09eccf014665e96f82e9ce1682be693242ef73a3421ca8f541a4b92: Status 404 returned error can't find the container with id 1d730c3cf09eccf014665e96f82e9ce1682be693242ef73a3421ca8f541a4b92
Jul 09 23:34:37.136987 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:37.136953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4" event={"ID":"32d804c5-f92b-4463-a737-eb395fe8af1a","Type":"ContainerStarted","Data":"1d730c3cf09eccf014665e96f82e9ce1682be693242ef73a3421ca8f541a4b92"}
Jul 09 23:34:39.901625 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:39.901591    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5xkp9"]
Jul 09 23:34:39.909118 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:39.909098    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xkp9"
Jul 09 23:34:39.912730 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:39.912707    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-b6k4g\""
Jul 09 23:34:39.913527 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:39.913502    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5xkp9"]
Jul 09 23:34:39.968281 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:39.968243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pst57\" (UniqueName: \"kubernetes.io/projected/981b54dd-b9f2-47e7-aa22-ae0cae9f38ff-kube-api-access-pst57\") pod \"authorino-operator-75cc8d5cb7-5xkp9\" (UID: \"981b54dd-b9f2-47e7-aa22-ae0cae9f38ff\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xkp9"
Jul 09 23:34:40.069672 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:40.069623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pst57\" (UniqueName: \"kubernetes.io/projected/981b54dd-b9f2-47e7-aa22-ae0cae9f38ff-kube-api-access-pst57\") pod \"authorino-operator-75cc8d5cb7-5xkp9\" (UID: \"981b54dd-b9f2-47e7-aa22-ae0cae9f38ff\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xkp9"
Jul 09 23:34:40.086561 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:40.086524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pst57\" (UniqueName: \"kubernetes.io/projected/981b54dd-b9f2-47e7-aa22-ae0cae9f38ff-kube-api-access-pst57\") pod \"authorino-operator-75cc8d5cb7-5xkp9\" (UID: \"981b54dd-b9f2-47e7-aa22-ae0cae9f38ff\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xkp9"
Jul 09 23:34:40.220594 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:40.220498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xkp9"
Jul 09 23:34:40.391298 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:40.391276    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5xkp9"]
Jul 09 23:34:40.393289 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:34:40.393258    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981b54dd_b9f2_47e7_aa22_ae0cae9f38ff.slice/crio-83e5583af042e3aa7dc6ac3e84ad5985be50397b52444c6ce6eb9ad530888d73 WatchSource:0}: Error finding container 83e5583af042e3aa7dc6ac3e84ad5985be50397b52444c6ce6eb9ad530888d73: Status 404 returned error can't find the container with id 83e5583af042e3aa7dc6ac3e84ad5985be50397b52444c6ce6eb9ad530888d73
Jul 09 23:34:41.163816 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:41.163756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xkp9" event={"ID":"981b54dd-b9f2-47e7-aa22-ae0cae9f38ff","Type":"ContainerStarted","Data":"83e5583af042e3aa7dc6ac3e84ad5985be50397b52444c6ce6eb9ad530888d73"}
Jul 09 23:34:42.172435 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:42.172399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4" event={"ID":"32d804c5-f92b-4463-a737-eb395fe8af1a","Type":"ContainerStarted","Data":"a583453021f46811b5feb0be0c4d8f2b339b23a18111b671a56e9530fbdb8b0d"}
Jul 09 23:34:42.172910 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:42.172519    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:42.194070 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:42.193999    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4" podStartSLOduration=0.799373491 podStartE2EDuration="6.193977671s" totalImagesPullingTime="5.39460418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:34:42.190911521 +0000 UTC m=+400.334842308" watchObservedRunningTime="2026-07-09 23:34:42.193977671 +0000 UTC m=+400.337908457"
Jul 09 23:34:44.125125 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:44.125046    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:44.185127 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:44.185090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xkp9" event={"ID":"981b54dd-b9f2-47e7-aa22-ae0cae9f38ff","Type":"ContainerStarted","Data":"6d55ef2d3e3d277317b4fd489088d25816e2a08ac90a24a6579ea2cf69b140e0"}
Jul 09 23:34:44.185289 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:44.185255    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xkp9"
Jul 09 23:34:44.200132 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:44.200087    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xkp9" podStartSLOduration=1.822017952 podStartE2EDuration="5.200071633s" totalImagesPullingTime="3.378053681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:34:44.199363751 +0000 UTC m=+402.343294539" watchObservedRunningTime="2026-07-09 23:34:44.200071633 +0000 UTC m=+402.344002419"
Jul 09 23:34:53.180490 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:53.180456    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:54.746010 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.745970    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"]
Jul 09 23:34:54.746427 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.746159    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" containerName="manager" containerID="cri-o://a583453021f46811b5feb0be0c4d8f2b339b23a18111b671a56e9530fbdb8b0d" gracePeriod=2
Jul 09 23:34:54.751172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.751144    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"]
Jul 09 23:34:54.761230 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.761193    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"]
Jul 09 23:34:54.761622 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.761593    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" containerName="manager" containerID="cri-o://fa8cbc628b5bcf5c9e39c55132ca19fcfbf44866d7126d2cb7992f31be1ff190" gracePeriod=2
Jul 09 23:34:54.767314 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.767287    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"]
Jul 09 23:34:54.767923 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.767901    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" containerName="manager"
Jul 09 23:34:54.768043 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.768033    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" containerName="manager"
Jul 09 23:34:54.768241 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.768230    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" containerName="manager"
Jul 09 23:34:54.771273 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:54.771190    2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-n4xxp\": pods \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:54.772933 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.772881    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"]
Jul 09 23:34:54.773038 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:54.772939    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:54.773038 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.772993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"
Jul 09 23:34:54.775079 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:54.775050    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:54.779898 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.779876    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gmhq7"]
Jul 09 23:34:54.780228 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.780211    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" containerName="manager"
Jul 09 23:34:54.780310 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.780232    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" containerName="manager"
Jul 09 23:34:54.780383 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.780372    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" containerName="manager"
Jul 09 23:34:54.783375 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.783354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gmhq7"
Jul 09 23:34:54.783909 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.783888    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"]
Jul 09 23:34:54.795105 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.795084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gmhq7"]
Jul 09 23:34:54.798308 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:54.798274    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:54.800240 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:54.800216    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:54.802127 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:54.802095    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:54.892235 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.892201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/563008eb-d990-43be-973e-7032304a94b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dzgb5\" (UID: \"563008eb-d990-43be-973e-7032304a94b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"
Jul 09 23:34:54.892407 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.892268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj78j\" (UniqueName: \"kubernetes.io/projected/7e3adeac-a2c0-48be-a349-6e70408d8058-kube-api-access-kj78j\") pod \"limitador-operator-controller-manager-859d7b78bd-gmhq7\" (UID: \"7e3adeac-a2c0-48be-a349-6e70408d8058\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gmhq7"
Jul 09 23:34:54.892407 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.892318    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckjf\" (UniqueName: \"kubernetes.io/projected/563008eb-d990-43be-973e-7032304a94b0-kube-api-access-2ckjf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dzgb5\" (UID: \"563008eb-d990-43be-973e-7032304a94b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"
Jul 09 23:34:54.995160 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.994137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj78j\" (UniqueName: \"kubernetes.io/projected/7e3adeac-a2c0-48be-a349-6e70408d8058-kube-api-access-kj78j\") pod \"limitador-operator-controller-manager-859d7b78bd-gmhq7\" (UID: \"7e3adeac-a2c0-48be-a349-6e70408d8058\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gmhq7"
Jul 09 23:34:54.995160 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.994204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckjf\" (UniqueName: \"kubernetes.io/projected/563008eb-d990-43be-973e-7032304a94b0-kube-api-access-2ckjf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dzgb5\" (UID: \"563008eb-d990-43be-973e-7032304a94b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"
Jul 09 23:34:54.995160 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.994283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/563008eb-d990-43be-973e-7032304a94b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dzgb5\" (UID: \"563008eb-d990-43be-973e-7032304a94b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"
Jul 09 23:34:54.995160 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:54.994750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/563008eb-d990-43be-973e-7032304a94b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dzgb5\" (UID: \"563008eb-d990-43be-973e-7032304a94b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"
Jul 09 23:34:55.009295 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.009270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckjf\" (UniqueName: \"kubernetes.io/projected/563008eb-d990-43be-973e-7032304a94b0-kube-api-access-2ckjf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dzgb5\" (UID: \"563008eb-d990-43be-973e-7032304a94b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"
Jul 09 23:34:55.013378 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.013352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj78j\" (UniqueName: \"kubernetes.io/projected/7e3adeac-a2c0-48be-a349-6e70408d8058-kube-api-access-kj78j\") pod \"limitador-operator-controller-manager-859d7b78bd-gmhq7\" (UID: \"7e3adeac-a2c0-48be-a349-6e70408d8058\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gmhq7"
Jul 09 23:34:55.021468 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.021441    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"]
Jul 09 23:34:55.024989 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.024970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"
Jul 09 23:34:55.027222 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.027194    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:55.029286 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.029256    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:55.030637 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.030617    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:55.034011 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.033991    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:55.039010 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.038986    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"]
Jul 09 23:34:55.068168 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.068130    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:55.070077 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.070054    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:55.095525 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.095492    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dqm8\" (UniqueName: \"kubernetes.io/projected/32d804c5-f92b-4463-a737-eb395fe8af1a-kube-api-access-6dqm8\") pod \"32d804c5-f92b-4463-a737-eb395fe8af1a\" (UID: \"32d804c5-f92b-4463-a737-eb395fe8af1a\") "
Jul 09 23:34:55.095722 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.095560    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbx8q\" (UniqueName: \"kubernetes.io/projected/f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645-kube-api-access-pbx8q\") pod \"f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645\" (UID: \"f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645\") "
Jul 09 23:34:55.095722 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.095596    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32d804c5-f92b-4463-a737-eb395fe8af1a-extensions-socket-volume\") pod \"32d804c5-f92b-4463-a737-eb395fe8af1a\" (UID: \"32d804c5-f92b-4463-a737-eb395fe8af1a\") "
Jul 09 23:34:55.095722 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.095704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ssv\" (UniqueName: \"kubernetes.io/projected/ed06cb77-5234-4427-ba27-c0e7a0026b02-kube-api-access-x6ssv\") pod \"kuadrant-operator-controller-manager-86878c769f-6hzhj\" (UID: \"ed06cb77-5234-4427-ba27-c0e7a0026b02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"
Jul 09 23:34:55.095855 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.095747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed06cb77-5234-4427-ba27-c0e7a0026b02-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6hzhj\" (UID: \"ed06cb77-5234-4427-ba27-c0e7a0026b02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"
Jul 09 23:34:55.096135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.096104    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d804c5-f92b-4463-a737-eb395fe8af1a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "32d804c5-f92b-4463-a737-eb395fe8af1a" (UID: "32d804c5-f92b-4463-a737-eb395fe8af1a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:34:55.097703 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.097671    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645-kube-api-access-pbx8q" (OuterVolumeSpecName: "kube-api-access-pbx8q") pod "f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" (UID: "f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645"). InnerVolumeSpecName "kube-api-access-pbx8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:34:55.097703 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.097674    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d804c5-f92b-4463-a737-eb395fe8af1a-kube-api-access-6dqm8" (OuterVolumeSpecName: "kube-api-access-6dqm8") pod "32d804c5-f92b-4463-a737-eb395fe8af1a" (UID: "32d804c5-f92b-4463-a737-eb395fe8af1a"). InnerVolumeSpecName "kube-api-access-6dqm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:34:55.175300 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.175259    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"
Jul 09 23:34:55.183361 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.183321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gmhq7"
Jul 09 23:34:55.195219 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.195196    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xkp9"
Jul 09 23:34:55.196857 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.196834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ssv\" (UniqueName: \"kubernetes.io/projected/ed06cb77-5234-4427-ba27-c0e7a0026b02-kube-api-access-x6ssv\") pod \"kuadrant-operator-controller-manager-86878c769f-6hzhj\" (UID: \"ed06cb77-5234-4427-ba27-c0e7a0026b02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"
Jul 09 23:34:55.196975 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.196870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed06cb77-5234-4427-ba27-c0e7a0026b02-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6hzhj\" (UID: \"ed06cb77-5234-4427-ba27-c0e7a0026b02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"
Jul 09 23:34:55.196975 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.196925    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dqm8\" (UniqueName: \"kubernetes.io/projected/32d804c5-f92b-4463-a737-eb395fe8af1a-kube-api-access-6dqm8\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:55.196975 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.196936    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbx8q\" (UniqueName: \"kubernetes.io/projected/f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645-kube-api-access-pbx8q\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:55.196975 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.196945    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32d804c5-f92b-4463-a737-eb395fe8af1a-extensions-socket-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:34:55.197356 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.197331    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:55.197356 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.197344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed06cb77-5234-4427-ba27-c0e7a0026b02-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6hzhj\" (UID: \"ed06cb77-5234-4427-ba27-c0e7a0026b02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"
Jul 09 23:34:55.199225 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.199198    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:55.207374 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.207351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ssv\" (UniqueName: \"kubernetes.io/projected/ed06cb77-5234-4427-ba27-c0e7a0026b02-kube-api-access-x6ssv\") pod \"kuadrant-operator-controller-manager-86878c769f-6hzhj\" (UID: \"ed06cb77-5234-4427-ba27-c0e7a0026b02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"
Jul 09 23:34:55.247842 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.245378    2580 generic.go:356] "Generic (PLEG): container finished" podID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" containerID="fa8cbc628b5bcf5c9e39c55132ca19fcfbf44866d7126d2cb7992f31be1ff190" exitCode=0
Jul 09 23:34:55.247842 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.245490    2580 scope.go:117] "RemoveContainer" containerID="fa8cbc628b5bcf5c9e39c55132ca19fcfbf44866d7126d2cb7992f31be1ff190"
Jul 09 23:34:55.247842 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.245695    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:55.251306 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.248905    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:55.251306 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.251161    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:55.257303 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.253477    2580 generic.go:356] "Generic (PLEG): container finished" podID="32d804c5-f92b-4463-a737-eb395fe8af1a" containerID="a583453021f46811b5feb0be0c4d8f2b339b23a18111b671a56e9530fbdb8b0d" exitCode=0
Jul 09 23:34:55.257303 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.253759    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:55.259108 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.259056    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:55.261581 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.261525    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:55.265553 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.265508    2580 scope.go:117] "RemoveContainer" containerID="fa8cbc628b5bcf5c9e39c55132ca19fcfbf44866d7126d2cb7992f31be1ff190"
Jul 09 23:34:55.270420 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.268310    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:55.270420 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.269954    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8cbc628b5bcf5c9e39c55132ca19fcfbf44866d7126d2cb7992f31be1ff190\": container with ID starting with fa8cbc628b5bcf5c9e39c55132ca19fcfbf44866d7126d2cb7992f31be1ff190 not found: ID does not exist" containerID="fa8cbc628b5bcf5c9e39c55132ca19fcfbf44866d7126d2cb7992f31be1ff190"
Jul 09 23:34:55.270420 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.270000    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8cbc628b5bcf5c9e39c55132ca19fcfbf44866d7126d2cb7992f31be1ff190"} err="failed to get container status \"fa8cbc628b5bcf5c9e39c55132ca19fcfbf44866d7126d2cb7992f31be1ff190\": rpc error: code = NotFound desc = could not find container \"fa8cbc628b5bcf5c9e39c55132ca19fcfbf44866d7126d2cb7992f31be1ff190\": container with ID starting with fa8cbc628b5bcf5c9e39c55132ca19fcfbf44866d7126d2cb7992f31be1ff190 not found: ID does not exist"
Jul 09 23:34:55.270420 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.270025    2580 scope.go:117] "RemoveContainer" containerID="a583453021f46811b5feb0be0c4d8f2b339b23a18111b671a56e9530fbdb8b0d"
Jul 09 23:34:55.270734 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.270492    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:55.282990 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.282962    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:55.284846 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.284813    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:55.288382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.288350    2580 scope.go:117] "RemoveContainer" containerID="a583453021f46811b5feb0be0c4d8f2b339b23a18111b671a56e9530fbdb8b0d"
Jul 09 23:34:55.288831 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:55.288786    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a583453021f46811b5feb0be0c4d8f2b339b23a18111b671a56e9530fbdb8b0d\": container with ID starting with a583453021f46811b5feb0be0c4d8f2b339b23a18111b671a56e9530fbdb8b0d not found: ID does not exist" containerID="a583453021f46811b5feb0be0c4d8f2b339b23a18111b671a56e9530fbdb8b0d"
Jul 09 23:34:55.288952 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.288842    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a583453021f46811b5feb0be0c4d8f2b339b23a18111b671a56e9530fbdb8b0d"} err="failed to get container status \"a583453021f46811b5feb0be0c4d8f2b339b23a18111b671a56e9530fbdb8b0d\": rpc error: code = NotFound desc = could not find container \"a583453021f46811b5feb0be0c4d8f2b339b23a18111b671a56e9530fbdb8b0d\": container with ID starting with a583453021f46811b5feb0be0c4d8f2b339b23a18111b671a56e9530fbdb8b0d not found: ID does not exist"
Jul 09 23:34:55.336606 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.336560    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"]
Jul 09 23:34:55.339472 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:34:55.339444    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563008eb_d990_43be_973e_7032304a94b0.slice/crio-7b757b10295c4ce53739839cf4b7cd61474059da8cafec6abd821401cf801231 WatchSource:0}: Error finding container 7b757b10295c4ce53739839cf4b7cd61474059da8cafec6abd821401cf801231: Status 404 returned error can't find the container with id 7b757b10295c4ce53739839cf4b7cd61474059da8cafec6abd821401cf801231
Jul 09 23:34:55.341399 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.341380    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"
Jul 09 23:34:55.392393 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.392364    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gmhq7"]
Jul 09 23:34:55.402516 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:34:55.399198    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3adeac_a2c0_48be_a349_6e70408d8058.slice/crio-b58a36b101b224f7b21b12c3cff276b15ba79d2551556d138255a6da6ce3bbeb WatchSource:0}: Error finding container b58a36b101b224f7b21b12c3cff276b15ba79d2551556d138255a6da6ce3bbeb: Status 404 returned error can't find the container with id b58a36b101b224f7b21b12c3cff276b15ba79d2551556d138255a6da6ce3bbeb
Jul 09 23:34:55.499270 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:55.499231    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"]
Jul 09 23:34:55.501737 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:34:55.501682    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded06cb77_5234_4427_ba27_c0e7a0026b02.slice/crio-7efa252a67af3f9735efdc335f974193d93915863dcf9244edd1d044feb9ede4 WatchSource:0}: Error finding container 7efa252a67af3f9735efdc335f974193d93915863dcf9244edd1d044feb9ede4: Status 404 returned error can't find the container with id 7efa252a67af3f9735efdc335f974193d93915863dcf9244edd1d044feb9ede4
Jul 09 23:34:56.261220 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.261185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj" event={"ID":"ed06cb77-5234-4427-ba27-c0e7a0026b02","Type":"ContainerStarted","Data":"05b56e0a8c1b7f189b320896e0ff27ca3bac54bdb7182a08d91a75948562e1ed"}
Jul 09 23:34:56.261676 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.261239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj" event={"ID":"ed06cb77-5234-4427-ba27-c0e7a0026b02","Type":"ContainerStarted","Data":"7efa252a67af3f9735efdc335f974193d93915863dcf9244edd1d044feb9ede4"}
Jul 09 23:34:56.261676 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.261300    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"
Jul 09 23:34:56.262892 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.262868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5" event={"ID":"563008eb-d990-43be-973e-7032304a94b0","Type":"ContainerStarted","Data":"f23a07d73c22683d30bcfb810afb92bf82ad1ea335b2a89e3a4c23063075e350"}
Jul 09 23:34:56.262995 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.262919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5" event={"ID":"563008eb-d990-43be-973e-7032304a94b0","Type":"ContainerStarted","Data":"7b757b10295c4ce53739839cf4b7cd61474059da8cafec6abd821401cf801231"}
Jul 09 23:34:56.262995 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.262941    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"
Jul 09 23:34:56.263213 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:56.263189    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:56.264445 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.264424    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gmhq7" event={"ID":"7e3adeac-a2c0-48be-a349-6e70408d8058","Type":"ContainerStarted","Data":"585da9e7922503060c914b727d850dd0147c49a43c82585596c68856ead5a338"}
Jul 09 23:34:56.264532 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.264467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gmhq7" event={"ID":"7e3adeac-a2c0-48be-a349-6e70408d8058","Type":"ContainerStarted","Data":"b58a36b101b224f7b21b12c3cff276b15ba79d2551556d138255a6da6ce3bbeb"}
Jul 09 23:34:56.264596 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.264528    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gmhq7"
Jul 09 23:34:56.264879 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:56.264855    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:56.285162 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.285109    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj" podStartSLOduration=1.285092277 podStartE2EDuration="1.285092277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:34:56.283423245 +0000 UTC m=+414.427354045" watchObservedRunningTime="2026-07-09 23:34:56.285092277 +0000 UTC m=+414.429023120"
Jul 09 23:34:56.285356 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:56.285317    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n4xxp\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n4xxp"
Jul 09 23:34:56.308736 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:34:56.308693    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6vpv4\" is forbidden: User \"system:node:ip-10-0-140-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-8.ec2.internal' and this object" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6vpv4"
Jul 09 23:34:56.308916 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.308871    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5" podStartSLOduration=2.30885308 podStartE2EDuration="2.30885308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:34:56.306908215 +0000 UTC m=+414.450839001" watchObservedRunningTime="2026-07-09 23:34:56.30885308 +0000 UTC m=+414.452783867"
Jul 09 23:34:56.328011 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.327951    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gmhq7" podStartSLOduration=2.327930265 podStartE2EDuration="2.327930265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:34:56.326106014 +0000 UTC m=+414.470036799" watchObservedRunningTime="2026-07-09 23:34:56.327930265 +0000 UTC m=+414.471861054"
Jul 09 23:34:56.419136 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.419103    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d804c5-f92b-4463-a737-eb395fe8af1a" path="/var/lib/kubelet/pods/32d804c5-f92b-4463-a737-eb395fe8af1a/volumes"
Jul 09 23:34:56.419446 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:34:56.419433    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645" path="/var/lib/kubelet/pods/f7b0f3c9-7eb5-4f88-84a6-fc1a19a52645/volumes"
Jul 09 23:35:07.274800 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:07.274763    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"
Jul 09 23:35:07.275327 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:07.274847    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"
Jul 09 23:35:07.275327 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:07.274884    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gmhq7"
Jul 09 23:35:07.343134 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:07.343092    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"]
Jul 09 23:35:07.343401 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:07.343370    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5" podUID="563008eb-d990-43be-973e-7032304a94b0" containerName="manager" containerID="cri-o://f23a07d73c22683d30bcfb810afb92bf82ad1ea335b2a89e3a4c23063075e350" gracePeriod=10
Jul 09 23:35:07.594552 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:07.594525    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"
Jul 09 23:35:07.701672 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:07.701609    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ckjf\" (UniqueName: \"kubernetes.io/projected/563008eb-d990-43be-973e-7032304a94b0-kube-api-access-2ckjf\") pod \"563008eb-d990-43be-973e-7032304a94b0\" (UID: \"563008eb-d990-43be-973e-7032304a94b0\") "
Jul 09 23:35:07.701836 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:07.701778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/563008eb-d990-43be-973e-7032304a94b0-extensions-socket-volume\") pod \"563008eb-d990-43be-973e-7032304a94b0\" (UID: \"563008eb-d990-43be-973e-7032304a94b0\") "
Jul 09 23:35:07.702160 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:07.702135    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563008eb-d990-43be-973e-7032304a94b0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "563008eb-d990-43be-973e-7032304a94b0" (UID: "563008eb-d990-43be-973e-7032304a94b0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:35:07.703794 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:07.703763    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563008eb-d990-43be-973e-7032304a94b0-kube-api-access-2ckjf" (OuterVolumeSpecName: "kube-api-access-2ckjf") pod "563008eb-d990-43be-973e-7032304a94b0" (UID: "563008eb-d990-43be-973e-7032304a94b0"). InnerVolumeSpecName "kube-api-access-2ckjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:35:07.803116 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:07.803071    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ckjf\" (UniqueName: \"kubernetes.io/projected/563008eb-d990-43be-973e-7032304a94b0-kube-api-access-2ckjf\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:35:07.803116 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:07.803108    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/563008eb-d990-43be-973e-7032304a94b0-extensions-socket-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:35:08.329926 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:08.329886    2580 generic.go:356] "Generic (PLEG): container finished" podID="563008eb-d990-43be-973e-7032304a94b0" containerID="f23a07d73c22683d30bcfb810afb92bf82ad1ea335b2a89e3a4c23063075e350" exitCode=0
Jul 09 23:35:08.330316 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:08.329938    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"
Jul 09 23:35:08.330316 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:08.329936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5" event={"ID":"563008eb-d990-43be-973e-7032304a94b0","Type":"ContainerDied","Data":"f23a07d73c22683d30bcfb810afb92bf82ad1ea335b2a89e3a4c23063075e350"}
Jul 09 23:35:08.330316 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:08.330052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5" event={"ID":"563008eb-d990-43be-973e-7032304a94b0","Type":"ContainerDied","Data":"7b757b10295c4ce53739839cf4b7cd61474059da8cafec6abd821401cf801231"}
Jul 09 23:35:08.330316 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:08.330081    2580 scope.go:117] "RemoveContainer" containerID="f23a07d73c22683d30bcfb810afb92bf82ad1ea335b2a89e3a4c23063075e350"
Jul 09 23:35:08.339072 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:08.339052    2580 scope.go:117] "RemoveContainer" containerID="f23a07d73c22683d30bcfb810afb92bf82ad1ea335b2a89e3a4c23063075e350"
Jul 09 23:35:08.339505 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:35:08.339487    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23a07d73c22683d30bcfb810afb92bf82ad1ea335b2a89e3a4c23063075e350\": container with ID starting with f23a07d73c22683d30bcfb810afb92bf82ad1ea335b2a89e3a4c23063075e350 not found: ID does not exist" containerID="f23a07d73c22683d30bcfb810afb92bf82ad1ea335b2a89e3a4c23063075e350"
Jul 09 23:35:08.339580 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:08.339518    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23a07d73c22683d30bcfb810afb92bf82ad1ea335b2a89e3a4c23063075e350"} err="failed to get container status \"f23a07d73c22683d30bcfb810afb92bf82ad1ea335b2a89e3a4c23063075e350\": rpc error: code = NotFound desc = could not find container \"f23a07d73c22683d30bcfb810afb92bf82ad1ea335b2a89e3a4c23063075e350\": container with ID starting with f23a07d73c22683d30bcfb810afb92bf82ad1ea335b2a89e3a4c23063075e350 not found: ID does not exist"
Jul 09 23:35:08.355113 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:08.355089    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"]
Jul 09 23:35:08.359276 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:08.359241    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dzgb5"]
Jul 09 23:35:08.418881 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:08.418850    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="563008eb-d990-43be-973e-7032304a94b0" path="/var/lib/kubelet/pods/563008eb-d990-43be-973e-7032304a94b0/volumes"
Jul 09 23:35:25.809075 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:25.808981    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"]
Jul 09 23:35:25.809562 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:25.809482    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563008eb-d990-43be-973e-7032304a94b0" containerName="manager"
Jul 09 23:35:25.809562 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:25.809502    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="563008eb-d990-43be-973e-7032304a94b0" containerName="manager"
Jul 09 23:35:25.809697 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:25.809582    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="563008eb-d990-43be-973e-7032304a94b0" containerName="manager"
Jul 09 23:35:25.811466 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:25.811438    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"
Jul 09 23:35:25.813786 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:25.813766    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xk9b9\""
Jul 09 23:35:25.813912 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:25.813765    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 09 23:35:25.819616 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:25.819590    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"]
Jul 09 23:35:25.899690 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:25.899645    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"]
Jul 09 23:35:25.959736 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:25.959701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb-config-file\") pod \"limitador-limitador-64c94fd64f-tmbgv\" (UID: \"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"
Jul 09 23:35:25.959902 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:25.959826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2fc\" (UniqueName: \"kubernetes.io/projected/7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb-kube-api-access-9t2fc\") pod \"limitador-limitador-64c94fd64f-tmbgv\" (UID: \"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"
Jul 09 23:35:26.060977 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.060893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2fc\" (UniqueName: \"kubernetes.io/projected/7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb-kube-api-access-9t2fc\") pod \"limitador-limitador-64c94fd64f-tmbgv\" (UID: \"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"
Jul 09 23:35:26.060977 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.060944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb-config-file\") pod \"limitador-limitador-64c94fd64f-tmbgv\" (UID: \"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"
Jul 09 23:35:26.061501 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.061482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb-config-file\") pod \"limitador-limitador-64c94fd64f-tmbgv\" (UID: \"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"
Jul 09 23:35:26.069865 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.069845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2fc\" (UniqueName: \"kubernetes.io/projected/7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb-kube-api-access-9t2fc\") pod \"limitador-limitador-64c94fd64f-tmbgv\" (UID: \"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"
Jul 09 23:35:26.121792 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.121758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"
Jul 09 23:35:26.254342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.254320    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"]
Jul 09 23:35:26.256186 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:35:26.256155    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7113d2f7_fb33_41d7_a6dc_43cb85dfcdfb.slice/crio-83ef336c98a49ee9b68770b6383d914b984e81b8402b52cc935656a038b9d85b WatchSource:0}: Error finding container 83ef336c98a49ee9b68770b6383d914b984e81b8402b52cc935656a038b9d85b: Status 404 returned error can't find the container with id 83ef336c98a49ee9b68770b6383d914b984e81b8402b52cc935656a038b9d85b
Jul 09 23:35:26.419179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.419093    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv" event={"ID":"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb","Type":"ContainerStarted","Data":"83ef336c98a49ee9b68770b6383d914b984e81b8402b52cc935656a038b9d85b"}
Jul 09 23:35:26.607738 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.607703    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-t7nkq"]
Jul 09 23:35:26.610358 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.610339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-t7nkq"
Jul 09 23:35:26.612427 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.612406    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rppgt\""
Jul 09 23:35:26.617332 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.617306    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-t7nkq"]
Jul 09 23:35:26.767084 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.767050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmxpj\" (UniqueName: \"kubernetes.io/projected/60a3269f-dbe7-4157-8d0c-01b74b01ee2d-kube-api-access-nmxpj\") pod \"authorino-58cd955675-t7nkq\" (UID: \"60a3269f-dbe7-4157-8d0c-01b74b01ee2d\") " pod="kuadrant-system/authorino-58cd955675-t7nkq"
Jul 09 23:35:26.804978 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.804944    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-77cj5"]
Jul 09 23:35:26.807078 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.807057    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-77cj5"
Jul 09 23:35:26.814493 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.814467    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-77cj5"]
Jul 09 23:35:26.868343 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.868312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmxpj\" (UniqueName: \"kubernetes.io/projected/60a3269f-dbe7-4157-8d0c-01b74b01ee2d-kube-api-access-nmxpj\") pod \"authorino-58cd955675-t7nkq\" (UID: \"60a3269f-dbe7-4157-8d0c-01b74b01ee2d\") " pod="kuadrant-system/authorino-58cd955675-t7nkq"
Jul 09 23:35:26.875980 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.875952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmxpj\" (UniqueName: \"kubernetes.io/projected/60a3269f-dbe7-4157-8d0c-01b74b01ee2d-kube-api-access-nmxpj\") pod \"authorino-58cd955675-t7nkq\" (UID: \"60a3269f-dbe7-4157-8d0c-01b74b01ee2d\") " pod="kuadrant-system/authorino-58cd955675-t7nkq"
Jul 09 23:35:26.920537 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.920502    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-t7nkq"
Jul 09 23:35:26.969972 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:26.969873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkxmk\" (UniqueName: \"kubernetes.io/projected/8f4330fc-6550-4310-b2e3-11a9339cdf04-kube-api-access-vkxmk\") pod \"authorino-75d556bd77-77cj5\" (UID: \"8f4330fc-6550-4310-b2e3-11a9339cdf04\") " pod="kuadrant-system/authorino-75d556bd77-77cj5"
Jul 09 23:35:27.072820 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:27.070971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkxmk\" (UniqueName: \"kubernetes.io/projected/8f4330fc-6550-4310-b2e3-11a9339cdf04-kube-api-access-vkxmk\") pod \"authorino-75d556bd77-77cj5\" (UID: \"8f4330fc-6550-4310-b2e3-11a9339cdf04\") " pod="kuadrant-system/authorino-75d556bd77-77cj5"
Jul 09 23:35:27.083443 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:27.082886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkxmk\" (UniqueName: \"kubernetes.io/projected/8f4330fc-6550-4310-b2e3-11a9339cdf04-kube-api-access-vkxmk\") pod \"authorino-75d556bd77-77cj5\" (UID: \"8f4330fc-6550-4310-b2e3-11a9339cdf04\") " pod="kuadrant-system/authorino-75d556bd77-77cj5"
Jul 09 23:35:27.101588 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:27.101522    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-t7nkq"]
Jul 09 23:35:27.105000 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:35:27.104946    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60a3269f_dbe7_4157_8d0c_01b74b01ee2d.slice/crio-943949b2b249992e703c5dd766ad04d48c8f7e3efb0cdd40b3a7ab0e783b90a1 WatchSource:0}: Error finding container 943949b2b249992e703c5dd766ad04d48c8f7e3efb0cdd40b3a7ab0e783b90a1: Status 404 returned error can't find the container with id 943949b2b249992e703c5dd766ad04d48c8f7e3efb0cdd40b3a7ab0e783b90a1
Jul 09 23:35:27.117609 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:27.117580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-77cj5"
Jul 09 23:35:27.305861 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:27.305805    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-77cj5"]
Jul 09 23:35:27.309096 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:35:27.309065    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f4330fc_6550_4310_b2e3_11a9339cdf04.slice/crio-dbb19a5271b17da68ad6915f588d00922f862fa40672081d71acda5f4a507e98 WatchSource:0}: Error finding container dbb19a5271b17da68ad6915f588d00922f862fa40672081d71acda5f4a507e98: Status 404 returned error can't find the container with id dbb19a5271b17da68ad6915f588d00922f862fa40672081d71acda5f4a507e98
Jul 09 23:35:27.428149 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:27.428105    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-t7nkq" event={"ID":"60a3269f-dbe7-4157-8d0c-01b74b01ee2d","Type":"ContainerStarted","Data":"943949b2b249992e703c5dd766ad04d48c8f7e3efb0cdd40b3a7ab0e783b90a1"}
Jul 09 23:35:27.429440 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:27.429411    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-77cj5" event={"ID":"8f4330fc-6550-4310-b2e3-11a9339cdf04","Type":"ContainerStarted","Data":"dbb19a5271b17da68ad6915f588d00922f862fa40672081d71acda5f4a507e98"}
Jul 09 23:35:29.452431 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:29.452134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv" event={"ID":"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb","Type":"ContainerStarted","Data":"95ff242e8aa3b114761b94e16f064fa2689c112735837abf024945b5c612a8ba"}
Jul 09 23:35:29.453088 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:29.452991    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"
Jul 09 23:35:29.473019 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:29.470623    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv" podStartSLOduration=1.428756837 podStartE2EDuration="4.470601298s" totalImagesPullingTime="3.041844461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:35:29.469052479 +0000 UTC m=+447.612983265" watchObservedRunningTime="2026-07-09 23:35:29.470601298 +0000 UTC m=+447.614532085"
Jul 09 23:35:32.470276 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:32.470230    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-77cj5" event={"ID":"8f4330fc-6550-4310-b2e3-11a9339cdf04","Type":"ContainerStarted","Data":"2a19367c594da343c7219e6cf6589ed6f19fd1f166c6082dd1f7b67cada2ffcb"}
Jul 09 23:35:32.471717 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:32.471694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-t7nkq" event={"ID":"60a3269f-dbe7-4157-8d0c-01b74b01ee2d","Type":"ContainerStarted","Data":"8acc87461240de76b10959da748d6198b262b3aa1c4d3e2bc0347b2e19a698a6"}
Jul 09 23:35:32.483850 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:32.483803    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-77cj5" podStartSLOduration=2.196360287 podStartE2EDuration="6.483791343s" totalImagesPullingTime="4.287431056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:35:32.48264815 +0000 UTC m=+450.626578941" watchObservedRunningTime="2026-07-09 23:35:32.483791343 +0000 UTC m=+450.627722128"
Jul 09 23:35:32.497124 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:32.497077    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-t7nkq" podStartSLOduration=2.017309414 podStartE2EDuration="6.497062032s" totalImagesPullingTime="4.479752618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:35:32.49515595 +0000 UTC m=+450.639086737" watchObservedRunningTime="2026-07-09 23:35:32.497062032 +0000 UTC m=+450.640992815"
Jul 09 23:35:32.523087 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:32.523057    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-t7nkq"]
Jul 09 23:35:34.480485 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:34.480437    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-t7nkq" podUID="60a3269f-dbe7-4157-8d0c-01b74b01ee2d" containerName="authorino" containerID="cri-o://8acc87461240de76b10959da748d6198b262b3aa1c4d3e2bc0347b2e19a698a6" gracePeriod=30
Jul 09 23:35:34.722690 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:34.722664    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-t7nkq"
Jul 09 23:35:34.736515 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:34.736446    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmxpj\" (UniqueName: \"kubernetes.io/projected/60a3269f-dbe7-4157-8d0c-01b74b01ee2d-kube-api-access-nmxpj\") pod \"60a3269f-dbe7-4157-8d0c-01b74b01ee2d\" (UID: \"60a3269f-dbe7-4157-8d0c-01b74b01ee2d\") "
Jul 09 23:35:34.738822 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:34.738786    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a3269f-dbe7-4157-8d0c-01b74b01ee2d-kube-api-access-nmxpj" (OuterVolumeSpecName: "kube-api-access-nmxpj") pod "60a3269f-dbe7-4157-8d0c-01b74b01ee2d" (UID: "60a3269f-dbe7-4157-8d0c-01b74b01ee2d"). InnerVolumeSpecName "kube-api-access-nmxpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:35:34.837383 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:34.837343    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmxpj\" (UniqueName: \"kubernetes.io/projected/60a3269f-dbe7-4157-8d0c-01b74b01ee2d-kube-api-access-nmxpj\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:35:35.486545 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:35.486511    2580 generic.go:356] "Generic (PLEG): container finished" podID="60a3269f-dbe7-4157-8d0c-01b74b01ee2d" containerID="8acc87461240de76b10959da748d6198b262b3aa1c4d3e2bc0347b2e19a698a6" exitCode=0
Jul 09 23:35:35.486998 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:35.486552    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-t7nkq"
Jul 09 23:35:35.486998 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:35.486578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-t7nkq" event={"ID":"60a3269f-dbe7-4157-8d0c-01b74b01ee2d","Type":"ContainerDied","Data":"8acc87461240de76b10959da748d6198b262b3aa1c4d3e2bc0347b2e19a698a6"}
Jul 09 23:35:35.486998 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:35.486608    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-t7nkq" event={"ID":"60a3269f-dbe7-4157-8d0c-01b74b01ee2d","Type":"ContainerDied","Data":"943949b2b249992e703c5dd766ad04d48c8f7e3efb0cdd40b3a7ab0e783b90a1"}
Jul 09 23:35:35.486998 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:35.486624    2580 scope.go:117] "RemoveContainer" containerID="8acc87461240de76b10959da748d6198b262b3aa1c4d3e2bc0347b2e19a698a6"
Jul 09 23:35:35.495099 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:35.495080    2580 scope.go:117] "RemoveContainer" containerID="8acc87461240de76b10959da748d6198b262b3aa1c4d3e2bc0347b2e19a698a6"
Jul 09 23:35:35.495362 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:35:35.495344    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8acc87461240de76b10959da748d6198b262b3aa1c4d3e2bc0347b2e19a698a6\": container with ID starting with 8acc87461240de76b10959da748d6198b262b3aa1c4d3e2bc0347b2e19a698a6 not found: ID does not exist" containerID="8acc87461240de76b10959da748d6198b262b3aa1c4d3e2bc0347b2e19a698a6"
Jul 09 23:35:35.495424 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:35.495370    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8acc87461240de76b10959da748d6198b262b3aa1c4d3e2bc0347b2e19a698a6"} err="failed to get container status \"8acc87461240de76b10959da748d6198b262b3aa1c4d3e2bc0347b2e19a698a6\": rpc error: code = NotFound desc = could not find container \"8acc87461240de76b10959da748d6198b262b3aa1c4d3e2bc0347b2e19a698a6\": container with ID starting with 8acc87461240de76b10959da748d6198b262b3aa1c4d3e2bc0347b2e19a698a6 not found: ID does not exist"
Jul 09 23:35:35.505578 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:35.505555    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-t7nkq"]
Jul 09 23:35:35.511044 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:35.511019    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-t7nkq"]
Jul 09 23:35:36.419130 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:36.419095    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60a3269f-dbe7-4157-8d0c-01b74b01ee2d" path="/var/lib/kubelet/pods/60a3269f-dbe7-4157-8d0c-01b74b01ee2d/volumes"
Jul 09 23:35:40.459939 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:40.459904    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"
Jul 09 23:35:40.774602 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:40.774566    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"]
Jul 09 23:35:40.774806 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:40.774760    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv" podUID="7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb" containerName="limitador" containerID="cri-o://95ff242e8aa3b114761b94e16f064fa2689c112735837abf024945b5c612a8ba" gracePeriod=30
Jul 09 23:35:41.518636 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:41.518600    2580 generic.go:356] "Generic (PLEG): container finished" podID="7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb" containerID="95ff242e8aa3b114761b94e16f064fa2689c112735837abf024945b5c612a8ba" exitCode=0
Jul 09 23:35:41.519038 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:41.518673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv" event={"ID":"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb","Type":"ContainerDied","Data":"95ff242e8aa3b114761b94e16f064fa2689c112735837abf024945b5c612a8ba"}
Jul 09 23:35:41.717894 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:41.717870    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"
Jul 09 23:35:41.788198 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:41.788105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t2fc\" (UniqueName: \"kubernetes.io/projected/7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb-kube-api-access-9t2fc\") pod \"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb\" (UID: \"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb\") "
Jul 09 23:35:41.788198 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:41.788179    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb-config-file\") pod \"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb\" (UID: \"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb\") "
Jul 09 23:35:41.788504 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:41.788482    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb-config-file" (OuterVolumeSpecName: "config-file") pod "7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb" (UID: "7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 23:35:41.790123 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:41.790104    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb-kube-api-access-9t2fc" (OuterVolumeSpecName: "kube-api-access-9t2fc") pod "7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb" (UID: "7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb"). InnerVolumeSpecName "kube-api-access-9t2fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:35:41.889013 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:41.888970    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t2fc\" (UniqueName: \"kubernetes.io/projected/7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb-kube-api-access-9t2fc\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:35:41.889013 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:41.889005    2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb-config-file\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:35:42.330997 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.330965    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj"]
Jul 09 23:35:42.331334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.331314    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb" containerName="limitador"
Jul 09 23:35:42.331334 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.331330    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb" containerName="limitador"
Jul 09 23:35:42.331527 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.331351    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a3269f-dbe7-4157-8d0c-01b74b01ee2d" containerName="authorino"
Jul 09 23:35:42.331527 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.331357    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a3269f-dbe7-4157-8d0c-01b74b01ee2d" containerName="authorino"
Jul 09 23:35:42.331527 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.331426    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a3269f-dbe7-4157-8d0c-01b74b01ee2d" containerName="authorino"
Jul 09 23:35:42.331527 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.331437    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb" containerName="limitador"
Jul 09 23:35:42.334446 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.334426    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj"
Jul 09 23:35:42.336921 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.336900    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 23:35:42.337792 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.337774    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kmvzs\""
Jul 09 23:35:42.337904 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.337777    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 23:35:42.337904 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.337812    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 09 23:35:42.342492 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.342470    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj"]
Jul 09 23:35:42.392427 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.392399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sflq7\" (UniqueName: \"kubernetes.io/projected/0dcc5167-3cad-4948-9e3e-50ebdb78ee5c-kube-api-access-sflq7\") pod \"postgres-69fb4594cf-fm4kj\" (UID: \"0dcc5167-3cad-4948-9e3e-50ebdb78ee5c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj"
Jul 09 23:35:42.392583 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.392448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0dcc5167-3cad-4948-9e3e-50ebdb78ee5c-data\") pod \"postgres-69fb4594cf-fm4kj\" (UID: \"0dcc5167-3cad-4948-9e3e-50ebdb78ee5c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj"
Jul 09 23:35:42.493254 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.493214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sflq7\" (UniqueName: \"kubernetes.io/projected/0dcc5167-3cad-4948-9e3e-50ebdb78ee5c-kube-api-access-sflq7\") pod \"postgres-69fb4594cf-fm4kj\" (UID: \"0dcc5167-3cad-4948-9e3e-50ebdb78ee5c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj"
Jul 09 23:35:42.493423 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.493275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0dcc5167-3cad-4948-9e3e-50ebdb78ee5c-data\") pod \"postgres-69fb4594cf-fm4kj\" (UID: \"0dcc5167-3cad-4948-9e3e-50ebdb78ee5c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj"
Jul 09 23:35:42.493625 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.493608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0dcc5167-3cad-4948-9e3e-50ebdb78ee5c-data\") pod \"postgres-69fb4594cf-fm4kj\" (UID: \"0dcc5167-3cad-4948-9e3e-50ebdb78ee5c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj"
Jul 09 23:35:42.501047 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.501018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sflq7\" (UniqueName: \"kubernetes.io/projected/0dcc5167-3cad-4948-9e3e-50ebdb78ee5c-kube-api-access-sflq7\") pod \"postgres-69fb4594cf-fm4kj\" (UID: \"0dcc5167-3cad-4948-9e3e-50ebdb78ee5c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj"
Jul 09 23:35:42.524383 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.524354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv" event={"ID":"7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb","Type":"ContainerDied","Data":"83ef336c98a49ee9b68770b6383d914b984e81b8402b52cc935656a038b9d85b"}
Jul 09 23:35:42.524383 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.524371    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"
Jul 09 23:35:42.524848 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.524410    2580 scope.go:117] "RemoveContainer" containerID="95ff242e8aa3b114761b94e16f064fa2689c112735837abf024945b5c612a8ba"
Jul 09 23:35:42.539841 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.539819    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"]
Jul 09 23:35:42.543377 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.543358    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tmbgv"]
Jul 09 23:35:42.646136 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.646045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj"
Jul 09 23:35:42.794008 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:42.791251    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj"]
Jul 09 23:35:43.531500 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:43.531452    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj" event={"ID":"0dcc5167-3cad-4948-9e3e-50ebdb78ee5c","Type":"ContainerStarted","Data":"18de481ad8b5df3ecffc1c62fdf0746f28cc6dbe27ec283b89c36e6d309314cc"}
Jul 09 23:35:44.420112 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:44.419992    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb" path="/var/lib/kubelet/pods/7113d2f7-fb33-41d7-a6dc-43cb85dfcdfb/volumes"
Jul 09 23:35:48.558186 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:48.558151    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj" event={"ID":"0dcc5167-3cad-4948-9e3e-50ebdb78ee5c","Type":"ContainerStarted","Data":"662fae7e4e8fc4503c8b95dce19544db710a376d8208fa86cdeee7a094a9b359"}
Jul 09 23:35:48.558570 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:48.558219    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj"
Jul 09 23:35:48.575131 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:48.575078    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj" podStartSLOduration=1.585104037 podStartE2EDuration="6.575060946s" totalImagesPullingTime="4.989956909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:35:48.572713247 +0000 UTC m=+466.716644032" watchObservedRunningTime="2026-07-09 23:35:48.575060946 +0000 UTC m=+466.718991732"
Jul 09 23:35:54.592629 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:54.592596    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fm4kj"
Jul 09 23:35:55.118455 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.118420    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pxcqd"]
Jul 09 23:35:55.124719 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.124692    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pxcqd"
Jul 09 23:35:55.128399 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.128366    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pxcqd"]
Jul 09 23:35:55.198173 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.198135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7txw\" (UniqueName: \"kubernetes.io/projected/b90009ad-8078-4876-93e7-f0e3c37c0b3f-kube-api-access-k7txw\") pod \"authorino-5d4b67c446-pxcqd\" (UID: \"b90009ad-8078-4876-93e7-f0e3c37c0b3f\") " pod="kuadrant-system/authorino-5d4b67c446-pxcqd"
Jul 09 23:35:55.299135 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.299094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7txw\" (UniqueName: \"kubernetes.io/projected/b90009ad-8078-4876-93e7-f0e3c37c0b3f-kube-api-access-k7txw\") pod \"authorino-5d4b67c446-pxcqd\" (UID: \"b90009ad-8078-4876-93e7-f0e3c37c0b3f\") " pod="kuadrant-system/authorino-5d4b67c446-pxcqd"
Jul 09 23:35:55.308640 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.308607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7txw\" (UniqueName: \"kubernetes.io/projected/b90009ad-8078-4876-93e7-f0e3c37c0b3f-kube-api-access-k7txw\") pod \"authorino-5d4b67c446-pxcqd\" (UID: \"b90009ad-8078-4876-93e7-f0e3c37c0b3f\") " pod="kuadrant-system/authorino-5d4b67c446-pxcqd"
Jul 09 23:35:55.340667 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.340629    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pxcqd"]
Jul 09 23:35:55.340891 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.340879    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pxcqd"
Jul 09 23:35:55.367593 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.367562    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6fb56bdb68-fc6r6"]
Jul 09 23:35:55.372511 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.372444    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fb56bdb68-fc6r6"
Jul 09 23:35:55.378477 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.378449    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fb56bdb68-fc6r6"]
Jul 09 23:35:55.451543 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.451506    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fb56bdb68-fc6r6"]
Jul 09 23:35:55.451862 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:35:55.451836    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tbl2j], unattached volumes=[], failed to process volumes=[kube-api-access-tbl2j]: context canceled" pod="kuadrant-system/authorino-6fb56bdb68-fc6r6" podUID="381a99d9-8df7-42b6-83c7-3ed42425a454"
Jul 09 23:35:55.474425 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.474394    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-699f96cf6-b8g59"]
Jul 09 23:35:55.477841 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.477819    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-699f96cf6-b8g59"
Jul 09 23:35:55.480166 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.480147    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 09 23:35:55.486045 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.486021    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-699f96cf6-b8g59"]
Jul 09 23:35:55.489913 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.489829    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pxcqd"]
Jul 09 23:35:55.493194 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:35:55.493170    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb90009ad_8078_4876_93e7_f0e3c37c0b3f.slice/crio-9c889a119f229fb5e291be0633dd54072ff236464b652c43c24e53ebb76b51a1 WatchSource:0}: Error finding container 9c889a119f229fb5e291be0633dd54072ff236464b652c43c24e53ebb76b51a1: Status 404 returned error can't find the container with id 9c889a119f229fb5e291be0633dd54072ff236464b652c43c24e53ebb76b51a1
Jul 09 23:35:55.500872 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.500840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbl2j\" (UniqueName: \"kubernetes.io/projected/381a99d9-8df7-42b6-83c7-3ed42425a454-kube-api-access-tbl2j\") pod \"authorino-6fb56bdb68-fc6r6\" (UID: \"381a99d9-8df7-42b6-83c7-3ed42425a454\") " pod="kuadrant-system/authorino-6fb56bdb68-fc6r6"
Jul 09 23:35:55.595518 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.595483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pxcqd" event={"ID":"b90009ad-8078-4876-93e7-f0e3c37c0b3f","Type":"ContainerStarted","Data":"9c889a119f229fb5e291be0633dd54072ff236464b652c43c24e53ebb76b51a1"}
Jul 09 23:35:55.595518 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.595507    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fb56bdb68-fc6r6"
Jul 09 23:35:55.601586 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.601561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbl2j\" (UniqueName: \"kubernetes.io/projected/381a99d9-8df7-42b6-83c7-3ed42425a454-kube-api-access-tbl2j\") pod \"authorino-6fb56bdb68-fc6r6\" (UID: \"381a99d9-8df7-42b6-83c7-3ed42425a454\") " pod="kuadrant-system/authorino-6fb56bdb68-fc6r6"
Jul 09 23:35:55.601734 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.601628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kls7\" (UniqueName: \"kubernetes.io/projected/dc4ec090-566e-4079-895a-f9f1e99bb686-kube-api-access-7kls7\") pod \"authorino-699f96cf6-b8g59\" (UID: \"dc4ec090-566e-4079-895a-f9f1e99bb686\") " pod="kuadrant-system/authorino-699f96cf6-b8g59"
Jul 09 23:35:55.601734 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.601725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc4ec090-566e-4079-895a-f9f1e99bb686-tls-cert\") pod \"authorino-699f96cf6-b8g59\" (UID: \"dc4ec090-566e-4079-895a-f9f1e99bb686\") " pod="kuadrant-system/authorino-699f96cf6-b8g59"
Jul 09 23:35:55.602329 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.602313    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fb56bdb68-fc6r6"
Jul 09 23:35:55.608703 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.608671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbl2j\" (UniqueName: \"kubernetes.io/projected/381a99d9-8df7-42b6-83c7-3ed42425a454-kube-api-access-tbl2j\") pod \"authorino-6fb56bdb68-fc6r6\" (UID: \"381a99d9-8df7-42b6-83c7-3ed42425a454\") " pod="kuadrant-system/authorino-6fb56bdb68-fc6r6"
Jul 09 23:35:55.702389 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.702307    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbl2j\" (UniqueName: \"kubernetes.io/projected/381a99d9-8df7-42b6-83c7-3ed42425a454-kube-api-access-tbl2j\") pod \"381a99d9-8df7-42b6-83c7-3ed42425a454\" (UID: \"381a99d9-8df7-42b6-83c7-3ed42425a454\") "
Jul 09 23:35:55.702535 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.702422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kls7\" (UniqueName: \"kubernetes.io/projected/dc4ec090-566e-4079-895a-f9f1e99bb686-kube-api-access-7kls7\") pod \"authorino-699f96cf6-b8g59\" (UID: \"dc4ec090-566e-4079-895a-f9f1e99bb686\") " pod="kuadrant-system/authorino-699f96cf6-b8g59"
Jul 09 23:35:55.702535 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.702459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc4ec090-566e-4079-895a-f9f1e99bb686-tls-cert\") pod \"authorino-699f96cf6-b8g59\" (UID: \"dc4ec090-566e-4079-895a-f9f1e99bb686\") " pod="kuadrant-system/authorino-699f96cf6-b8g59"
Jul 09 23:35:55.704404 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.704375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381a99d9-8df7-42b6-83c7-3ed42425a454-kube-api-access-tbl2j" (OuterVolumeSpecName: "kube-api-access-tbl2j") pod "381a99d9-8df7-42b6-83c7-3ed42425a454" (UID: "381a99d9-8df7-42b6-83c7-3ed42425a454"). InnerVolumeSpecName "kube-api-access-tbl2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:35:55.704833 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.704814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc4ec090-566e-4079-895a-f9f1e99bb686-tls-cert\") pod \"authorino-699f96cf6-b8g59\" (UID: \"dc4ec090-566e-4079-895a-f9f1e99bb686\") " pod="kuadrant-system/authorino-699f96cf6-b8g59"
Jul 09 23:35:55.709807 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.709786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kls7\" (UniqueName: \"kubernetes.io/projected/dc4ec090-566e-4079-895a-f9f1e99bb686-kube-api-access-7kls7\") pod \"authorino-699f96cf6-b8g59\" (UID: \"dc4ec090-566e-4079-895a-f9f1e99bb686\") " pod="kuadrant-system/authorino-699f96cf6-b8g59"
Jul 09 23:35:55.787119 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.787082    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-699f96cf6-b8g59"
Jul 09 23:35:55.803382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.803344    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbl2j\" (UniqueName: \"kubernetes.io/projected/381a99d9-8df7-42b6-83c7-3ed42425a454-kube-api-access-tbl2j\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:35:55.916027 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:55.915964    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-699f96cf6-b8g59"]
Jul 09 23:35:55.918033 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:35:55.918011    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc4ec090_566e_4079_895a_f9f1e99bb686.slice/crio-9bfcc7a9d400cbbac6900d4d84e6af4e59ddb364ffffa8c3f9ccd592bf6419fd WatchSource:0}: Error finding container 9bfcc7a9d400cbbac6900d4d84e6af4e59ddb364ffffa8c3f9ccd592bf6419fd: Status 404 returned error can't find the container with id 9bfcc7a9d400cbbac6900d4d84e6af4e59ddb364ffffa8c3f9ccd592bf6419fd
Jul 09 23:35:56.603005 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:56.602909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-699f96cf6-b8g59" event={"ID":"dc4ec090-566e-4079-895a-f9f1e99bb686","Type":"ContainerStarted","Data":"82cdcff42d8b5b7f846711d773f91fdfff93c310718a0752c682dbc941f13d9e"}
Jul 09 23:35:56.603005 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:56.602952    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-699f96cf6-b8g59" event={"ID":"dc4ec090-566e-4079-895a-f9f1e99bb686","Type":"ContainerStarted","Data":"9bfcc7a9d400cbbac6900d4d84e6af4e59ddb364ffffa8c3f9ccd592bf6419fd"}
Jul 09 23:35:56.604500 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:56.604474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pxcqd" event={"ID":"b90009ad-8078-4876-93e7-f0e3c37c0b3f","Type":"ContainerStarted","Data":"a5ec1190a5dfe405669a0aef1cb5943dc31944b168116736d7de8cb2b898e58a"}
Jul 09 23:35:56.604634 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:56.604487    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fb56bdb68-fc6r6"
Jul 09 23:35:56.604634 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:56.604530    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-pxcqd" podUID="b90009ad-8078-4876-93e7-f0e3c37c0b3f" containerName="authorino" containerID="cri-o://a5ec1190a5dfe405669a0aef1cb5943dc31944b168116736d7de8cb2b898e58a" gracePeriod=30
Jul 09 23:35:56.618260 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:56.618215    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-699f96cf6-b8g59" podStartSLOduration=1.086595883 podStartE2EDuration="1.61819824s" totalImagesPullingTime="531.602357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:35:56.616087204 +0000 UTC m=+474.760017991" watchObservedRunningTime="2026-07-09 23:35:56.61819824 +0000 UTC m=+474.762129026"
Jul 09 23:35:56.639043 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:56.639008    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fb56bdb68-fc6r6"]
Jul 09 23:35:56.647302 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:56.647272    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6fb56bdb68-fc6r6"]
Jul 09 23:35:56.650322 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:56.650291    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-77cj5"]
Jul 09 23:35:56.650534 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:56.650506    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-77cj5" podUID="8f4330fc-6550-4310-b2e3-11a9339cdf04" containerName="authorino" containerID="cri-o://2a19367c594da343c7219e6cf6589ed6f19fd1f166c6082dd1f7b67cada2ffcb" gracePeriod=30
Jul 09 23:35:56.676136 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:56.676073    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-pxcqd" podStartSLOduration=1.235144836 podStartE2EDuration="1.676051202s" totalImagesPullingTime="440.906366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:35:56.656434072 +0000 UTC m=+474.800364863" watchObservedRunningTime="2026-07-09 23:35:56.676051202 +0000 UTC m=+474.819981989"
Jul 09 23:35:56.898956 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:56.898927    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pxcqd"
Jul 09 23:35:56.941493 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:56.941468    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-77cj5"
Jul 09 23:35:57.015086 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.015056    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkxmk\" (UniqueName: \"kubernetes.io/projected/8f4330fc-6550-4310-b2e3-11a9339cdf04-kube-api-access-vkxmk\") pod \"8f4330fc-6550-4310-b2e3-11a9339cdf04\" (UID: \"8f4330fc-6550-4310-b2e3-11a9339cdf04\") "
Jul 09 23:35:57.015275 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.015098    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7txw\" (UniqueName: \"kubernetes.io/projected/b90009ad-8078-4876-93e7-f0e3c37c0b3f-kube-api-access-k7txw\") pod \"b90009ad-8078-4876-93e7-f0e3c37c0b3f\" (UID: \"b90009ad-8078-4876-93e7-f0e3c37c0b3f\") "
Jul 09 23:35:57.017129 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.017095    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4330fc-6550-4310-b2e3-11a9339cdf04-kube-api-access-vkxmk" (OuterVolumeSpecName: "kube-api-access-vkxmk") pod "8f4330fc-6550-4310-b2e3-11a9339cdf04" (UID: "8f4330fc-6550-4310-b2e3-11a9339cdf04"). InnerVolumeSpecName "kube-api-access-vkxmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:35:57.017129 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.017106    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90009ad-8078-4876-93e7-f0e3c37c0b3f-kube-api-access-k7txw" (OuterVolumeSpecName: "kube-api-access-k7txw") pod "b90009ad-8078-4876-93e7-f0e3c37c0b3f" (UID: "b90009ad-8078-4876-93e7-f0e3c37c0b3f"). InnerVolumeSpecName "kube-api-access-k7txw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:35:57.116735 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.116615    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkxmk\" (UniqueName: \"kubernetes.io/projected/8f4330fc-6550-4310-b2e3-11a9339cdf04-kube-api-access-vkxmk\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:35:57.116735 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.116647    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7txw\" (UniqueName: \"kubernetes.io/projected/b90009ad-8078-4876-93e7-f0e3c37c0b3f-kube-api-access-k7txw\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:35:57.611218 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.611177    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f4330fc-6550-4310-b2e3-11a9339cdf04" containerID="2a19367c594da343c7219e6cf6589ed6f19fd1f166c6082dd1f7b67cada2ffcb" exitCode=0
Jul 09 23:35:57.611698 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.611231    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-77cj5"
Jul 09 23:35:57.611698 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.611268    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-77cj5" event={"ID":"8f4330fc-6550-4310-b2e3-11a9339cdf04","Type":"ContainerDied","Data":"2a19367c594da343c7219e6cf6589ed6f19fd1f166c6082dd1f7b67cada2ffcb"}
Jul 09 23:35:57.611698 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.611315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-77cj5" event={"ID":"8f4330fc-6550-4310-b2e3-11a9339cdf04","Type":"ContainerDied","Data":"dbb19a5271b17da68ad6915f588d00922f862fa40672081d71acda5f4a507e98"}
Jul 09 23:35:57.611698 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.611332    2580 scope.go:117] "RemoveContainer" containerID="2a19367c594da343c7219e6cf6589ed6f19fd1f166c6082dd1f7b67cada2ffcb"
Jul 09 23:35:57.612759 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.612740    2580 generic.go:356] "Generic (PLEG): container finished" podID="b90009ad-8078-4876-93e7-f0e3c37c0b3f" containerID="a5ec1190a5dfe405669a0aef1cb5943dc31944b168116736d7de8cb2b898e58a" exitCode=0
Jul 09 23:35:57.612834 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.612798    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pxcqd"
Jul 09 23:35:57.612894 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.612833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pxcqd" event={"ID":"b90009ad-8078-4876-93e7-f0e3c37c0b3f","Type":"ContainerDied","Data":"a5ec1190a5dfe405669a0aef1cb5943dc31944b168116736d7de8cb2b898e58a"}
Jul 09 23:35:57.612894 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.612870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pxcqd" event={"ID":"b90009ad-8078-4876-93e7-f0e3c37c0b3f","Type":"ContainerDied","Data":"9c889a119f229fb5e291be0633dd54072ff236464b652c43c24e53ebb76b51a1"}
Jul 09 23:35:57.621057 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.621037    2580 scope.go:117] "RemoveContainer" containerID="2a19367c594da343c7219e6cf6589ed6f19fd1f166c6082dd1f7b67cada2ffcb"
Jul 09 23:35:57.621366 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:35:57.621338    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a19367c594da343c7219e6cf6589ed6f19fd1f166c6082dd1f7b67cada2ffcb\": container with ID starting with 2a19367c594da343c7219e6cf6589ed6f19fd1f166c6082dd1f7b67cada2ffcb not found: ID does not exist" containerID="2a19367c594da343c7219e6cf6589ed6f19fd1f166c6082dd1f7b67cada2ffcb"
Jul 09 23:35:57.621450 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.621376    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a19367c594da343c7219e6cf6589ed6f19fd1f166c6082dd1f7b67cada2ffcb"} err="failed to get container status \"2a19367c594da343c7219e6cf6589ed6f19fd1f166c6082dd1f7b67cada2ffcb\": rpc error: code = NotFound desc = could not find container \"2a19367c594da343c7219e6cf6589ed6f19fd1f166c6082dd1f7b67cada2ffcb\": container with ID starting with 2a19367c594da343c7219e6cf6589ed6f19fd1f166c6082dd1f7b67cada2ffcb not found: ID does not exist"
Jul 09 23:35:57.621450 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.621399    2580 scope.go:117] "RemoveContainer" containerID="a5ec1190a5dfe405669a0aef1cb5943dc31944b168116736d7de8cb2b898e58a"
Jul 09 23:35:57.629320 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.629294    2580 scope.go:117] "RemoveContainer" containerID="a5ec1190a5dfe405669a0aef1cb5943dc31944b168116736d7de8cb2b898e58a"
Jul 09 23:35:57.629593 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:35:57.629574    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5ec1190a5dfe405669a0aef1cb5943dc31944b168116736d7de8cb2b898e58a\": container with ID starting with a5ec1190a5dfe405669a0aef1cb5943dc31944b168116736d7de8cb2b898e58a not found: ID does not exist" containerID="a5ec1190a5dfe405669a0aef1cb5943dc31944b168116736d7de8cb2b898e58a"
Jul 09 23:35:57.629638 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.629603    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5ec1190a5dfe405669a0aef1cb5943dc31944b168116736d7de8cb2b898e58a"} err="failed to get container status \"a5ec1190a5dfe405669a0aef1cb5943dc31944b168116736d7de8cb2b898e58a\": rpc error: code = NotFound desc = could not find container \"a5ec1190a5dfe405669a0aef1cb5943dc31944b168116736d7de8cb2b898e58a\": container with ID starting with a5ec1190a5dfe405669a0aef1cb5943dc31944b168116736d7de8cb2b898e58a not found: ID does not exist"
Jul 09 23:35:57.632764 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.632645    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-77cj5"]
Jul 09 23:35:57.635885 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.635860    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-77cj5"]
Jul 09 23:35:57.644596 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.644568    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pxcqd"]
Jul 09 23:35:57.647157 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:57.647132    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pxcqd"]
Jul 09 23:35:58.419731 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:58.419697    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="381a99d9-8df7-42b6-83c7-3ed42425a454" path="/var/lib/kubelet/pods/381a99d9-8df7-42b6-83c7-3ed42425a454/volumes"
Jul 09 23:35:58.420007 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:58.419990    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f4330fc-6550-4310-b2e3-11a9339cdf04" path="/var/lib/kubelet/pods/8f4330fc-6550-4310-b2e3-11a9339cdf04/volumes"
Jul 09 23:35:58.420324 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:35:58.420308    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b90009ad-8078-4876-93e7-f0e3c37c0b3f" path="/var/lib/kubelet/pods/b90009ad-8078-4876-93e7-f0e3c37c0b3f/volumes"
Jul 09 23:36:00.806767 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.806729    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56c846cb99-2z4hh"]
Jul 09 23:36:00.807240 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.807213    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b90009ad-8078-4876-93e7-f0e3c37c0b3f" containerName="authorino"
Jul 09 23:36:00.807240 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.807232    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b90009ad-8078-4876-93e7-f0e3c37c0b3f" containerName="authorino"
Jul 09 23:36:00.807363 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.807244    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4330fc-6550-4310-b2e3-11a9339cdf04" containerName="authorino"
Jul 09 23:36:00.807363 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.807254    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4330fc-6550-4310-b2e3-11a9339cdf04" containerName="authorino"
Jul 09 23:36:00.807459 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.807371    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b90009ad-8078-4876-93e7-f0e3c37c0b3f" containerName="authorino"
Jul 09 23:36:00.807459 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.807385    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f4330fc-6550-4310-b2e3-11a9339cdf04" containerName="authorino"
Jul 09 23:36:00.886727 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.886694    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56c846cb99-2z4hh"]
Jul 09 23:36:00.886923 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.886776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56c846cb99-2z4hh"
Jul 09 23:36:00.889666 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.889598    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 09 23:36:00.889815 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.889603    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 09 23:36:00.889815 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.889604    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kbmvl\""
Jul 09 23:36:00.948901 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.948868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5684z\" (UniqueName: \"kubernetes.io/projected/8d33f5b0-ee5f-4715-98e8-14a0e102419d-kube-api-access-5684z\") pod \"maas-controller-56c846cb99-2z4hh\" (UID: \"8d33f5b0-ee5f-4715-98e8-14a0e102419d\") " pod="opendatahub/maas-controller-56c846cb99-2z4hh"
Jul 09 23:36:00.949041 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.948913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d33f5b0-ee5f-4715-98e8-14a0e102419d-webhook-cert\") pod \"maas-controller-56c846cb99-2z4hh\" (UID: \"8d33f5b0-ee5f-4715-98e8-14a0e102419d\") " pod="opendatahub/maas-controller-56c846cb99-2z4hh"
Jul 09 23:36:00.981839 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.981807    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-849b8469f6-v2bgh"]
Jul 09 23:36:00.999065 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.999003    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-849b8469f6-v2bgh"]
Jul 09 23:36:00.999233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:00.999099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-849b8469f6-v2bgh"
Jul 09 23:36:01.050223 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.050184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrc5q\" (UniqueName: \"kubernetes.io/projected/efffe372-198b-4aaf-bb86-55f33ea647f6-kube-api-access-vrc5q\") pod \"maas-controller-849b8469f6-v2bgh\" (UID: \"efffe372-198b-4aaf-bb86-55f33ea647f6\") " pod="opendatahub/maas-controller-849b8469f6-v2bgh"
Jul 09 23:36:01.050415 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.050254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efffe372-198b-4aaf-bb86-55f33ea647f6-webhook-cert\") pod \"maas-controller-849b8469f6-v2bgh\" (UID: \"efffe372-198b-4aaf-bb86-55f33ea647f6\") " pod="opendatahub/maas-controller-849b8469f6-v2bgh"
Jul 09 23:36:01.050415 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.050294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5684z\" (UniqueName: \"kubernetes.io/projected/8d33f5b0-ee5f-4715-98e8-14a0e102419d-kube-api-access-5684z\") pod \"maas-controller-56c846cb99-2z4hh\" (UID: \"8d33f5b0-ee5f-4715-98e8-14a0e102419d\") " pod="opendatahub/maas-controller-56c846cb99-2z4hh"
Jul 09 23:36:01.050415 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.050323    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d33f5b0-ee5f-4715-98e8-14a0e102419d-webhook-cert\") pod \"maas-controller-56c846cb99-2z4hh\" (UID: \"8d33f5b0-ee5f-4715-98e8-14a0e102419d\") " pod="opendatahub/maas-controller-56c846cb99-2z4hh"
Jul 09 23:36:01.052931 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.052898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d33f5b0-ee5f-4715-98e8-14a0e102419d-webhook-cert\") pod \"maas-controller-56c846cb99-2z4hh\" (UID: \"8d33f5b0-ee5f-4715-98e8-14a0e102419d\") " pod="opendatahub/maas-controller-56c846cb99-2z4hh"
Jul 09 23:36:01.063091 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.063023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5684z\" (UniqueName: \"kubernetes.io/projected/8d33f5b0-ee5f-4715-98e8-14a0e102419d-kube-api-access-5684z\") pod \"maas-controller-56c846cb99-2z4hh\" (UID: \"8d33f5b0-ee5f-4715-98e8-14a0e102419d\") " pod="opendatahub/maas-controller-56c846cb99-2z4hh"
Jul 09 23:36:01.151119 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.151076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrc5q\" (UniqueName: \"kubernetes.io/projected/efffe372-198b-4aaf-bb86-55f33ea647f6-kube-api-access-vrc5q\") pod \"maas-controller-849b8469f6-v2bgh\" (UID: \"efffe372-198b-4aaf-bb86-55f33ea647f6\") " pod="opendatahub/maas-controller-849b8469f6-v2bgh"
Jul 09 23:36:01.151310 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.151136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efffe372-198b-4aaf-bb86-55f33ea647f6-webhook-cert\") pod \"maas-controller-849b8469f6-v2bgh\" (UID: \"efffe372-198b-4aaf-bb86-55f33ea647f6\") " pod="opendatahub/maas-controller-849b8469f6-v2bgh"
Jul 09 23:36:01.153583 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.153557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efffe372-198b-4aaf-bb86-55f33ea647f6-webhook-cert\") pod \"maas-controller-849b8469f6-v2bgh\" (UID: \"efffe372-198b-4aaf-bb86-55f33ea647f6\") " pod="opendatahub/maas-controller-849b8469f6-v2bgh"
Jul 09 23:36:01.170094 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.170064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrc5q\" (UniqueName: \"kubernetes.io/projected/efffe372-198b-4aaf-bb86-55f33ea647f6-kube-api-access-vrc5q\") pod \"maas-controller-849b8469f6-v2bgh\" (UID: \"efffe372-198b-4aaf-bb86-55f33ea647f6\") " pod="opendatahub/maas-controller-849b8469f6-v2bgh"
Jul 09 23:36:01.197963 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.197932    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56c846cb99-2z4hh"
Jul 09 23:36:01.309511 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.309450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-849b8469f6-v2bgh"
Jul 09 23:36:01.393759 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.393732    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56c846cb99-2z4hh"]
Jul 09 23:36:01.395745 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:36:01.395715    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d33f5b0_ee5f_4715_98e8_14a0e102419d.slice/crio-f0374da0f91f963860097bc517345c7cef1d3b1ca58302de6848755aa15e5fdb WatchSource:0}: Error finding container f0374da0f91f963860097bc517345c7cef1d3b1ca58302de6848755aa15e5fdb: Status 404 returned error can't find the container with id f0374da0f91f963860097bc517345c7cef1d3b1ca58302de6848755aa15e5fdb
Jul 09 23:36:01.455076 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.455050    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-849b8469f6-v2bgh"]
Jul 09 23:36:01.457141 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:36:01.457112    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefffe372_198b_4aaf_bb86_55f33ea647f6.slice/crio-997a3a06a94ba202e45b5737a420aaf9402c8cd1bc1b2791939701bbbee96dc1 WatchSource:0}: Error finding container 997a3a06a94ba202e45b5737a420aaf9402c8cd1bc1b2791939701bbbee96dc1: Status 404 returned error can't find the container with id 997a3a06a94ba202e45b5737a420aaf9402c8cd1bc1b2791939701bbbee96dc1
Jul 09 23:36:01.635445 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.635353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-849b8469f6-v2bgh" event={"ID":"efffe372-198b-4aaf-bb86-55f33ea647f6","Type":"ContainerStarted","Data":"997a3a06a94ba202e45b5737a420aaf9402c8cd1bc1b2791939701bbbee96dc1"}
Jul 09 23:36:01.636513 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:01.636490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56c846cb99-2z4hh" event={"ID":"8d33f5b0-ee5f-4715-98e8-14a0e102419d","Type":"ContainerStarted","Data":"f0374da0f91f963860097bc517345c7cef1d3b1ca58302de6848755aa15e5fdb"}
Jul 09 23:36:04.109133 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:04.109107    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 09 23:36:04.659510 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:04.659473    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56c846cb99-2z4hh" event={"ID":"8d33f5b0-ee5f-4715-98e8-14a0e102419d","Type":"ContainerStarted","Data":"2bc8ac087c2e10e689e2fe9720b5455ac22275b72b8fcfb3c62b5e552b6c8759"}
Jul 09 23:36:04.659745 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:04.659617    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56c846cb99-2z4hh"
Jul 09 23:36:04.661275 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:04.661250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-849b8469f6-v2bgh" event={"ID":"efffe372-198b-4aaf-bb86-55f33ea647f6","Type":"ContainerStarted","Data":"2a8601157368b9d69d39213b280796af655f42a2aa395aa8f315fd346f632e9e"}
Jul 09 23:36:04.661402 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:04.661370    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-849b8469f6-v2bgh"
Jul 09 23:36:04.676784 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:04.676733    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56c846cb99-2z4hh" podStartSLOduration=1.9624178140000001 podStartE2EDuration="4.676718645s" totalImagesPullingTime="2.714300831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:36:04.676109704 +0000 UTC m=+482.820040489" watchObservedRunningTime="2026-07-09 23:36:04.676718645 +0000 UTC m=+482.820649430"
Jul 09 23:36:04.692970 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:04.692924    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-849b8469f6-v2bgh" podStartSLOduration=2.044759129 podStartE2EDuration="4.692909961s" totalImagesPullingTime="2.648150832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:36:04.692526547 +0000 UTC m=+482.836457332" watchObservedRunningTime="2026-07-09 23:36:04.692909961 +0000 UTC m=+482.836840747"
Jul 09 23:36:15.671158 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:15.671128    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56c846cb99-2z4hh"
Jul 09 23:36:15.671876 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:15.671354    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-849b8469f6-v2bgh"
Jul 09 23:36:15.728141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:15.728100    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56c846cb99-2z4hh"]
Jul 09 23:36:15.728356 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:15.728330    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-56c846cb99-2z4hh" podUID="8d33f5b0-ee5f-4715-98e8-14a0e102419d" containerName="manager" containerID="cri-o://2bc8ac087c2e10e689e2fe9720b5455ac22275b72b8fcfb3c62b5e552b6c8759" gracePeriod=30
Jul 09 23:36:15.978874 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:15.978846    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56c846cb99-2z4hh"
Jul 09 23:36:16.072989 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.072948    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5684z\" (UniqueName: \"kubernetes.io/projected/8d33f5b0-ee5f-4715-98e8-14a0e102419d-kube-api-access-5684z\") pod \"8d33f5b0-ee5f-4715-98e8-14a0e102419d\" (UID: \"8d33f5b0-ee5f-4715-98e8-14a0e102419d\") "
Jul 09 23:36:16.073162 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.073038    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d33f5b0-ee5f-4715-98e8-14a0e102419d-webhook-cert\") pod \"8d33f5b0-ee5f-4715-98e8-14a0e102419d\" (UID: \"8d33f5b0-ee5f-4715-98e8-14a0e102419d\") "
Jul 09 23:36:16.075288 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.075249    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d33f5b0-ee5f-4715-98e8-14a0e102419d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8d33f5b0-ee5f-4715-98e8-14a0e102419d" (UID: "8d33f5b0-ee5f-4715-98e8-14a0e102419d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 23:36:16.075392 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.075253    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d33f5b0-ee5f-4715-98e8-14a0e102419d-kube-api-access-5684z" (OuterVolumeSpecName: "kube-api-access-5684z") pod "8d33f5b0-ee5f-4715-98e8-14a0e102419d" (UID: "8d33f5b0-ee5f-4715-98e8-14a0e102419d"). InnerVolumeSpecName "kube-api-access-5684z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:36:16.173826 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.173779    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d33f5b0-ee5f-4715-98e8-14a0e102419d-webhook-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:36:16.173826 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.173821    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5684z\" (UniqueName: \"kubernetes.io/projected/8d33f5b0-ee5f-4715-98e8-14a0e102419d-kube-api-access-5684z\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:36:16.722355 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.722271    2580 generic.go:356] "Generic (PLEG): container finished" podID="8d33f5b0-ee5f-4715-98e8-14a0e102419d" containerID="2bc8ac087c2e10e689e2fe9720b5455ac22275b72b8fcfb3c62b5e552b6c8759" exitCode=0
Jul 09 23:36:16.722355 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.722329    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56c846cb99-2z4hh"
Jul 09 23:36:16.722355 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.722335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56c846cb99-2z4hh" event={"ID":"8d33f5b0-ee5f-4715-98e8-14a0e102419d","Type":"ContainerDied","Data":"2bc8ac087c2e10e689e2fe9720b5455ac22275b72b8fcfb3c62b5e552b6c8759"}
Jul 09 23:36:16.722843 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.722420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56c846cb99-2z4hh" event={"ID":"8d33f5b0-ee5f-4715-98e8-14a0e102419d","Type":"ContainerDied","Data":"f0374da0f91f963860097bc517345c7cef1d3b1ca58302de6848755aa15e5fdb"}
Jul 09 23:36:16.722843 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.722443    2580 scope.go:117] "RemoveContainer" containerID="2bc8ac087c2e10e689e2fe9720b5455ac22275b72b8fcfb3c62b5e552b6c8759"
Jul 09 23:36:16.731144 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.730997    2580 scope.go:117] "RemoveContainer" containerID="2bc8ac087c2e10e689e2fe9720b5455ac22275b72b8fcfb3c62b5e552b6c8759"
Jul 09 23:36:16.731264 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:36:16.731245    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc8ac087c2e10e689e2fe9720b5455ac22275b72b8fcfb3c62b5e552b6c8759\": container with ID starting with 2bc8ac087c2e10e689e2fe9720b5455ac22275b72b8fcfb3c62b5e552b6c8759 not found: ID does not exist" containerID="2bc8ac087c2e10e689e2fe9720b5455ac22275b72b8fcfb3c62b5e552b6c8759"
Jul 09 23:36:16.731311 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.731279    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc8ac087c2e10e689e2fe9720b5455ac22275b72b8fcfb3c62b5e552b6c8759"} err="failed to get container status \"2bc8ac087c2e10e689e2fe9720b5455ac22275b72b8fcfb3c62b5e552b6c8759\": rpc error: code = NotFound desc = could not find container \"2bc8ac087c2e10e689e2fe9720b5455ac22275b72b8fcfb3c62b5e552b6c8759\": container with ID starting with 2bc8ac087c2e10e689e2fe9720b5455ac22275b72b8fcfb3c62b5e552b6c8759 not found: ID does not exist"
Jul 09 23:36:16.737931 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.737905    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56c846cb99-2z4hh"]
Jul 09 23:36:16.739975 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:16.739954    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56c846cb99-2z4hh"]
Jul 09 23:36:18.419363 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:18.419332    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d33f5b0-ee5f-4715-98e8-14a0e102419d" path="/var/lib/kubelet/pods/8d33f5b0-ee5f-4715-98e8-14a0e102419d/volumes"
Jul 09 23:36:38.799312 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:38.799267    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"]
Jul 09 23:36:38.799791 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:38.799648    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d33f5b0-ee5f-4715-98e8-14a0e102419d" containerName="manager"
Jul 09 23:36:38.799791 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:38.799675    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d33f5b0-ee5f-4715-98e8-14a0e102419d" containerName="manager"
Jul 09 23:36:38.799791 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:38.799745    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d33f5b0-ee5f-4715-98e8-14a0e102419d" containerName="manager"
Jul 09 23:36:38.802528 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:38.802512    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"
Jul 09 23:36:38.804995 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:38.804971    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 09 23:36:38.805121 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:38.805021    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fvtlc\""
Jul 09 23:36:38.813470 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:38.813443    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"]
Jul 09 23:36:38.961910 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:38.961864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59cb762-a576-4ddb-b0c8-cf44efba25b1-openshift-service-ca\") pod \"maas-api-f64fcbf8c-4x9q2\" (UID: \"c59cb762-a576-4ddb-b0c8-cf44efba25b1\") " pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"
Jul 09 23:36:38.962095 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:38.961988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4ndb\" (UniqueName: \"kubernetes.io/projected/c59cb762-a576-4ddb-b0c8-cf44efba25b1-kube-api-access-l4ndb\") pod \"maas-api-f64fcbf8c-4x9q2\" (UID: \"c59cb762-a576-4ddb-b0c8-cf44efba25b1\") " pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"
Jul 09 23:36:38.962095 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:38.962062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c59cb762-a576-4ddb-b0c8-cf44efba25b1-maas-api-tls\") pod \"maas-api-f64fcbf8c-4x9q2\" (UID: \"c59cb762-a576-4ddb-b0c8-cf44efba25b1\") " pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"
Jul 09 23:36:39.062565 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.062471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ndb\" (UniqueName: \"kubernetes.io/projected/c59cb762-a576-4ddb-b0c8-cf44efba25b1-kube-api-access-l4ndb\") pod \"maas-api-f64fcbf8c-4x9q2\" (UID: \"c59cb762-a576-4ddb-b0c8-cf44efba25b1\") " pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"
Jul 09 23:36:39.062565 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.062523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c59cb762-a576-4ddb-b0c8-cf44efba25b1-maas-api-tls\") pod \"maas-api-f64fcbf8c-4x9q2\" (UID: \"c59cb762-a576-4ddb-b0c8-cf44efba25b1\") " pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"
Jul 09 23:36:39.062565 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.062558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59cb762-a576-4ddb-b0c8-cf44efba25b1-openshift-service-ca\") pod \"maas-api-f64fcbf8c-4x9q2\" (UID: \"c59cb762-a576-4ddb-b0c8-cf44efba25b1\") " pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"
Jul 09 23:36:39.063221 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.063202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59cb762-a576-4ddb-b0c8-cf44efba25b1-openshift-service-ca\") pod \"maas-api-f64fcbf8c-4x9q2\" (UID: \"c59cb762-a576-4ddb-b0c8-cf44efba25b1\") " pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"
Jul 09 23:36:39.064994 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.064969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c59cb762-a576-4ddb-b0c8-cf44efba25b1-maas-api-tls\") pod \"maas-api-f64fcbf8c-4x9q2\" (UID: \"c59cb762-a576-4ddb-b0c8-cf44efba25b1\") " pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"
Jul 09 23:36:39.070179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.070153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4ndb\" (UniqueName: \"kubernetes.io/projected/c59cb762-a576-4ddb-b0c8-cf44efba25b1-kube-api-access-l4ndb\") pod \"maas-api-f64fcbf8c-4x9q2\" (UID: \"c59cb762-a576-4ddb-b0c8-cf44efba25b1\") " pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"
Jul 09 23:36:39.114037 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.114002    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"
Jul 09 23:36:39.252744 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.252721    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"]
Jul 09 23:36:39.254966 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:36:39.254932    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59cb762_a576_4ddb_b0c8_cf44efba25b1.slice/crio-27a76b799baca3bc254901ac8894d23c322e5687c922efe5c38191f154dbefd6 WatchSource:0}: Error finding container 27a76b799baca3bc254901ac8894d23c322e5687c922efe5c38191f154dbefd6: Status 404 returned error can't find the container with id 27a76b799baca3bc254901ac8894d23c322e5687c922efe5c38191f154dbefd6
Jul 09 23:36:39.403299 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.403202    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-6kzrr"]
Jul 09 23:36:39.407078 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.407048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-6kzrr"
Jul 09 23:36:39.409587 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.409560    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 09 23:36:39.409734 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.409586    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kxkxz\""
Jul 09 23:36:39.415820 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.415796    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-6kzrr"]
Jul 09 23:36:39.566634 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.566594    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd99b0f-9531-4af7-9e2e-32009c97048c-plugins-config-volume\") pod \"payload-processing-58f86c7798-6kzrr\" (UID: \"fdd99b0f-9531-4af7-9e2e-32009c97048c\") " pod="openshift-ingress/payload-processing-58f86c7798-6kzrr"
Jul 09 23:36:39.566806 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.566720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h45tk\" (UniqueName: \"kubernetes.io/projected/fdd99b0f-9531-4af7-9e2e-32009c97048c-kube-api-access-h45tk\") pod \"payload-processing-58f86c7798-6kzrr\" (UID: \"fdd99b0f-9531-4af7-9e2e-32009c97048c\") " pod="openshift-ingress/payload-processing-58f86c7798-6kzrr"
Jul 09 23:36:39.667538 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.667450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h45tk\" (UniqueName: \"kubernetes.io/projected/fdd99b0f-9531-4af7-9e2e-32009c97048c-kube-api-access-h45tk\") pod \"payload-processing-58f86c7798-6kzrr\" (UID: \"fdd99b0f-9531-4af7-9e2e-32009c97048c\") " pod="openshift-ingress/payload-processing-58f86c7798-6kzrr"
Jul 09 23:36:39.667538 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.667525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd99b0f-9531-4af7-9e2e-32009c97048c-plugins-config-volume\") pod \"payload-processing-58f86c7798-6kzrr\" (UID: \"fdd99b0f-9531-4af7-9e2e-32009c97048c\") " pod="openshift-ingress/payload-processing-58f86c7798-6kzrr"
Jul 09 23:36:39.668330 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.668306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd99b0f-9531-4af7-9e2e-32009c97048c-plugins-config-volume\") pod \"payload-processing-58f86c7798-6kzrr\" (UID: \"fdd99b0f-9531-4af7-9e2e-32009c97048c\") " pod="openshift-ingress/payload-processing-58f86c7798-6kzrr"
Jul 09 23:36:39.675081 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.675054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h45tk\" (UniqueName: \"kubernetes.io/projected/fdd99b0f-9531-4af7-9e2e-32009c97048c-kube-api-access-h45tk\") pod \"payload-processing-58f86c7798-6kzrr\" (UID: \"fdd99b0f-9531-4af7-9e2e-32009c97048c\") " pod="openshift-ingress/payload-processing-58f86c7798-6kzrr"
Jul 09 23:36:39.719479 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.719439    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-6kzrr"
Jul 09 23:36:39.838490 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.838449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2" event={"ID":"c59cb762-a576-4ddb-b0c8-cf44efba25b1","Type":"ContainerStarted","Data":"27a76b799baca3bc254901ac8894d23c322e5687c922efe5c38191f154dbefd6"}
Jul 09 23:36:39.862907 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:39.862877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-6kzrr"]
Jul 09 23:36:39.864464 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:36:39.864436    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd99b0f_9531_4af7_9e2e_32009c97048c.slice/crio-cfc432defcbefcfb92201021311f108578009f7b5b8d84c26c43362b1c602ed3 WatchSource:0}: Error finding container cfc432defcbefcfb92201021311f108578009f7b5b8d84c26c43362b1c602ed3: Status 404 returned error can't find the container with id cfc432defcbefcfb92201021311f108578009f7b5b8d84c26c43362b1c602ed3
Jul 09 23:36:40.846514 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:40.846475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-6kzrr" event={"ID":"fdd99b0f-9531-4af7-9e2e-32009c97048c","Type":"ContainerStarted","Data":"cfc432defcbefcfb92201021311f108578009f7b5b8d84c26c43362b1c602ed3"}
Jul 09 23:36:41.854494 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:41.854454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2" event={"ID":"c59cb762-a576-4ddb-b0c8-cf44efba25b1","Type":"ContainerStarted","Data":"0e2d6e525b9670ce7beb998fe206a52fcab9ca3141b23a352af1566eea63ad0e"}
Jul 09 23:36:41.855001 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:41.854570    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"
Jul 09 23:36:41.856019 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:41.855994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-6kzrr" event={"ID":"fdd99b0f-9531-4af7-9e2e-32009c97048c","Type":"ContainerStarted","Data":"89048afb5825b7f3bdd2592ffe44505e722479080c6c6d95a5a73d75422b8735"}
Jul 09 23:36:41.856131 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:41.856061    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-6kzrr"
Jul 09 23:36:41.870206 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:41.870160    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2" podStartSLOduration=1.999140411 podStartE2EDuration="3.870145595s" totalImagesPullingTime="1.871005184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:36:41.868958077 +0000 UTC m=+520.012888876" watchObservedRunningTime="2026-07-09 23:36:41.870145595 +0000 UTC m=+520.014076381"
Jul 09 23:36:41.884409 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:41.884364    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-6kzrr" podStartSLOduration=1.6221557089999998 podStartE2EDuration="2.884347566s" totalImagesPullingTime="1.262191857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:36:41.882038273 +0000 UTC m=+520.025969057" watchObservedRunningTime="2026-07-09 23:36:41.884347566 +0000 UTC m=+520.028278351"
Jul 09 23:36:47.866417 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:47.866385    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f64fcbf8c-4x9q2"
Jul 09 23:36:52.863620 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:36:52.863590    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-6kzrr"
Jul 09 23:38:48.406806 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:48.406763    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b55548785-pmhm9"]
Jul 09 23:38:48.410301 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:48.410278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b55548785-pmhm9"
Jul 09 23:38:48.421680 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:48.421636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b55548785-pmhm9"]
Jul 09 23:38:48.451312 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:48.451272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/85f4e702-49a2-42d8-a08f-63f9242654c2-tls-cert\") pod \"authorino-6b55548785-pmhm9\" (UID: \"85f4e702-49a2-42d8-a08f-63f9242654c2\") " pod="kuadrant-system/authorino-6b55548785-pmhm9"
Jul 09 23:38:48.451449 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:48.451357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmhfh\" (UniqueName: \"kubernetes.io/projected/85f4e702-49a2-42d8-a08f-63f9242654c2-kube-api-access-bmhfh\") pod \"authorino-6b55548785-pmhm9\" (UID: \"85f4e702-49a2-42d8-a08f-63f9242654c2\") " pod="kuadrant-system/authorino-6b55548785-pmhm9"
Jul 09 23:38:48.551803 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:48.551773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmhfh\" (UniqueName: \"kubernetes.io/projected/85f4e702-49a2-42d8-a08f-63f9242654c2-kube-api-access-bmhfh\") pod \"authorino-6b55548785-pmhm9\" (UID: \"85f4e702-49a2-42d8-a08f-63f9242654c2\") " pod="kuadrant-system/authorino-6b55548785-pmhm9"
Jul 09 23:38:48.551803 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:48.551810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/85f4e702-49a2-42d8-a08f-63f9242654c2-tls-cert\") pod \"authorino-6b55548785-pmhm9\" (UID: \"85f4e702-49a2-42d8-a08f-63f9242654c2\") " pod="kuadrant-system/authorino-6b55548785-pmhm9"
Jul 09 23:38:48.554112 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:48.554090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/85f4e702-49a2-42d8-a08f-63f9242654c2-tls-cert\") pod \"authorino-6b55548785-pmhm9\" (UID: \"85f4e702-49a2-42d8-a08f-63f9242654c2\") " pod="kuadrant-system/authorino-6b55548785-pmhm9"
Jul 09 23:38:48.558987 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:48.558965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmhfh\" (UniqueName: \"kubernetes.io/projected/85f4e702-49a2-42d8-a08f-63f9242654c2-kube-api-access-bmhfh\") pod \"authorino-6b55548785-pmhm9\" (UID: \"85f4e702-49a2-42d8-a08f-63f9242654c2\") " pod="kuadrant-system/authorino-6b55548785-pmhm9"
Jul 09 23:38:48.721431 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:48.721352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b55548785-pmhm9"
Jul 09 23:38:48.857862 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:48.857837    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b55548785-pmhm9"]
Jul 09 23:38:48.859920 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:38:48.859869    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85f4e702_49a2_42d8_a08f_63f9242654c2.slice/crio-1b7cc59d0797bf0172d86d2e6e259003b80ad0e659af72f712f4358473eb12f7 WatchSource:0}: Error finding container 1b7cc59d0797bf0172d86d2e6e259003b80ad0e659af72f712f4358473eb12f7: Status 404 returned error can't find the container with id 1b7cc59d0797bf0172d86d2e6e259003b80ad0e659af72f712f4358473eb12f7
Jul 09 23:38:48.861314 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:48.861289    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 23:38:49.510270 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:49.510225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b55548785-pmhm9" event={"ID":"85f4e702-49a2-42d8-a08f-63f9242654c2","Type":"ContainerStarted","Data":"49e6765732da1623ec5518832ede0caca12efb58ddc4d68f32498ad9bae93a1c"}
Jul 09 23:38:49.510270 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:49.510270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b55548785-pmhm9" event={"ID":"85f4e702-49a2-42d8-a08f-63f9242654c2","Type":"ContainerStarted","Data":"1b7cc59d0797bf0172d86d2e6e259003b80ad0e659af72f712f4358473eb12f7"}
Jul 09 23:38:49.527674 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:49.527590    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b55548785-pmhm9" podStartSLOduration=1.107536323 podStartE2EDuration="1.527568834s" totalImagesPullingTime="420.032511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:38:49.527279006 +0000 UTC m=+647.671209794" watchObservedRunningTime="2026-07-09 23:38:49.527568834 +0000 UTC m=+647.671499622"
Jul 09 23:38:49.556716 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:49.556615    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-699f96cf6-b8g59"]
Jul 09 23:38:49.557127 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:49.557093    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-699f96cf6-b8g59" podUID="dc4ec090-566e-4079-895a-f9f1e99bb686" containerName="authorino" containerID="cri-o://82cdcff42d8b5b7f846711d773f91fdfff93c310718a0752c682dbc941f13d9e" gracePeriod=30
Jul 09 23:38:49.827859 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:49.827829    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-699f96cf6-b8g59"
Jul 09 23:38:49.862349 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:49.862313    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc4ec090-566e-4079-895a-f9f1e99bb686-tls-cert\") pod \"dc4ec090-566e-4079-895a-f9f1e99bb686\" (UID: \"dc4ec090-566e-4079-895a-f9f1e99bb686\") "
Jul 09 23:38:49.862530 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:49.862399    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kls7\" (UniqueName: \"kubernetes.io/projected/dc4ec090-566e-4079-895a-f9f1e99bb686-kube-api-access-7kls7\") pod \"dc4ec090-566e-4079-895a-f9f1e99bb686\" (UID: \"dc4ec090-566e-4079-895a-f9f1e99bb686\") "
Jul 09 23:38:49.864370 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:49.864337    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4ec090-566e-4079-895a-f9f1e99bb686-kube-api-access-7kls7" (OuterVolumeSpecName: "kube-api-access-7kls7") pod "dc4ec090-566e-4079-895a-f9f1e99bb686" (UID: "dc4ec090-566e-4079-895a-f9f1e99bb686"). InnerVolumeSpecName "kube-api-access-7kls7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:38:49.875923 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:49.875895    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc4ec090-566e-4079-895a-f9f1e99bb686-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "dc4ec090-566e-4079-895a-f9f1e99bb686" (UID: "dc4ec090-566e-4079-895a-f9f1e99bb686"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 23:38:49.963627 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:49.963581    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc4ec090-566e-4079-895a-f9f1e99bb686-tls-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:38:49.963627 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:49.963623    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kls7\" (UniqueName: \"kubernetes.io/projected/dc4ec090-566e-4079-895a-f9f1e99bb686-kube-api-access-7kls7\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:38:50.517214 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:50.517177    2580 generic.go:356] "Generic (PLEG): container finished" podID="dc4ec090-566e-4079-895a-f9f1e99bb686" containerID="82cdcff42d8b5b7f846711d773f91fdfff93c310718a0752c682dbc941f13d9e" exitCode=0
Jul 09 23:38:50.517615 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:50.517217    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-699f96cf6-b8g59"
Jul 09 23:38:50.517615 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:50.517267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-699f96cf6-b8g59" event={"ID":"dc4ec090-566e-4079-895a-f9f1e99bb686","Type":"ContainerDied","Data":"82cdcff42d8b5b7f846711d773f91fdfff93c310718a0752c682dbc941f13d9e"}
Jul 09 23:38:50.517615 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:50.517319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-699f96cf6-b8g59" event={"ID":"dc4ec090-566e-4079-895a-f9f1e99bb686","Type":"ContainerDied","Data":"9bfcc7a9d400cbbac6900d4d84e6af4e59ddb364ffffa8c3f9ccd592bf6419fd"}
Jul 09 23:38:50.517615 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:50.517345    2580 scope.go:117] "RemoveContainer" containerID="82cdcff42d8b5b7f846711d773f91fdfff93c310718a0752c682dbc941f13d9e"
Jul 09 23:38:50.526121 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:50.526099    2580 scope.go:117] "RemoveContainer" containerID="82cdcff42d8b5b7f846711d773f91fdfff93c310718a0752c682dbc941f13d9e"
Jul 09 23:38:50.526426 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:38:50.526404    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82cdcff42d8b5b7f846711d773f91fdfff93c310718a0752c682dbc941f13d9e\": container with ID starting with 82cdcff42d8b5b7f846711d773f91fdfff93c310718a0752c682dbc941f13d9e not found: ID does not exist" containerID="82cdcff42d8b5b7f846711d773f91fdfff93c310718a0752c682dbc941f13d9e"
Jul 09 23:38:50.526504 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:50.526439    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cdcff42d8b5b7f846711d773f91fdfff93c310718a0752c682dbc941f13d9e"} err="failed to get container status \"82cdcff42d8b5b7f846711d773f91fdfff93c310718a0752c682dbc941f13d9e\": rpc error: code = NotFound desc = could not find container \"82cdcff42d8b5b7f846711d773f91fdfff93c310718a0752c682dbc941f13d9e\": container with ID starting with 82cdcff42d8b5b7f846711d773f91fdfff93c310718a0752c682dbc941f13d9e not found: ID does not exist"
Jul 09 23:38:50.532888 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:50.532857    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-699f96cf6-b8g59"]
Jul 09 23:38:50.536179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:50.536146    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-699f96cf6-b8g59"]
Jul 09 23:38:52.421142 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:38:52.421108    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc4ec090-566e-4079-895a-f9f1e99bb686" path="/var/lib/kubelet/pods/dc4ec090-566e-4079-895a-f9f1e99bb686/volumes"
Jul 09 23:40:00.467165 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.467076    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-849b8469f6-v2bgh"]
Jul 09 23:40:00.467743 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.467357    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-849b8469f6-v2bgh" podUID="efffe372-198b-4aaf-bb86-55f33ea647f6" containerName="manager" containerID="cri-o://2a8601157368b9d69d39213b280796af655f42a2aa395aa8f315fd346f632e9e" gracePeriod=30
Jul 09 23:40:00.718425 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.718358    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-849b8469f6-v2bgh"
Jul 09 23:40:00.760063 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.760024    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efffe372-198b-4aaf-bb86-55f33ea647f6-webhook-cert\") pod \"efffe372-198b-4aaf-bb86-55f33ea647f6\" (UID: \"efffe372-198b-4aaf-bb86-55f33ea647f6\") "
Jul 09 23:40:00.760227 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.760144    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrc5q\" (UniqueName: \"kubernetes.io/projected/efffe372-198b-4aaf-bb86-55f33ea647f6-kube-api-access-vrc5q\") pod \"efffe372-198b-4aaf-bb86-55f33ea647f6\" (UID: \"efffe372-198b-4aaf-bb86-55f33ea647f6\") "
Jul 09 23:40:00.762321 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.762281    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efffe372-198b-4aaf-bb86-55f33ea647f6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "efffe372-198b-4aaf-bb86-55f33ea647f6" (UID: "efffe372-198b-4aaf-bb86-55f33ea647f6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 23:40:00.762455 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.762363    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efffe372-198b-4aaf-bb86-55f33ea647f6-kube-api-access-vrc5q" (OuterVolumeSpecName: "kube-api-access-vrc5q") pod "efffe372-198b-4aaf-bb86-55f33ea647f6" (UID: "efffe372-198b-4aaf-bb86-55f33ea647f6"). InnerVolumeSpecName "kube-api-access-vrc5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:40:00.861694 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.861633    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrc5q\" (UniqueName: \"kubernetes.io/projected/efffe372-198b-4aaf-bb86-55f33ea647f6-kube-api-access-vrc5q\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:40:00.861694 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.861687    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efffe372-198b-4aaf-bb86-55f33ea647f6-webhook-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:40:00.882086 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.882056    2580 generic.go:356] "Generic (PLEG): container finished" podID="efffe372-198b-4aaf-bb86-55f33ea647f6" containerID="2a8601157368b9d69d39213b280796af655f42a2aa395aa8f315fd346f632e9e" exitCode=0
Jul 09 23:40:00.882272 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.882098    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-849b8469f6-v2bgh"
Jul 09 23:40:00.882272 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.882141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-849b8469f6-v2bgh" event={"ID":"efffe372-198b-4aaf-bb86-55f33ea647f6","Type":"ContainerDied","Data":"2a8601157368b9d69d39213b280796af655f42a2aa395aa8f315fd346f632e9e"}
Jul 09 23:40:00.882272 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.882202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-849b8469f6-v2bgh" event={"ID":"efffe372-198b-4aaf-bb86-55f33ea647f6","Type":"ContainerDied","Data":"997a3a06a94ba202e45b5737a420aaf9402c8cd1bc1b2791939701bbbee96dc1"}
Jul 09 23:40:00.882272 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.882234    2580 scope.go:117] "RemoveContainer" containerID="2a8601157368b9d69d39213b280796af655f42a2aa395aa8f315fd346f632e9e"
Jul 09 23:40:00.892116 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.892098    2580 scope.go:117] "RemoveContainer" containerID="2a8601157368b9d69d39213b280796af655f42a2aa395aa8f315fd346f632e9e"
Jul 09 23:40:00.892424 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:40:00.892405    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8601157368b9d69d39213b280796af655f42a2aa395aa8f315fd346f632e9e\": container with ID starting with 2a8601157368b9d69d39213b280796af655f42a2aa395aa8f315fd346f632e9e not found: ID does not exist" containerID="2a8601157368b9d69d39213b280796af655f42a2aa395aa8f315fd346f632e9e"
Jul 09 23:40:00.892479 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.892434    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8601157368b9d69d39213b280796af655f42a2aa395aa8f315fd346f632e9e"} err="failed to get container status \"2a8601157368b9d69d39213b280796af655f42a2aa395aa8f315fd346f632e9e\": rpc error: code = NotFound desc = could not find container \"2a8601157368b9d69d39213b280796af655f42a2aa395aa8f315fd346f632e9e\": container with ID starting with 2a8601157368b9d69d39213b280796af655f42a2aa395aa8f315fd346f632e9e not found: ID does not exist"
Jul 09 23:40:00.902640 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.902615    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-849b8469f6-v2bgh"]
Jul 09 23:40:00.906069 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:00.906047    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-849b8469f6-v2bgh"]
Jul 09 23:40:01.667742 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.667703    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-849b8469f6-m5d4c"]
Jul 09 23:40:01.668254 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.668234    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc4ec090-566e-4079-895a-f9f1e99bb686" containerName="authorino"
Jul 09 23:40:01.668327 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.668258    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc4ec090-566e-4079-895a-f9f1e99bb686" containerName="authorino"
Jul 09 23:40:01.668327 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.668271    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efffe372-198b-4aaf-bb86-55f33ea647f6" containerName="manager"
Jul 09 23:40:01.668327 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.668279    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="efffe372-198b-4aaf-bb86-55f33ea647f6" containerName="manager"
Jul 09 23:40:01.668460 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.668370    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="efffe372-198b-4aaf-bb86-55f33ea647f6" containerName="manager"
Jul 09 23:40:01.668460 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.668385    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc4ec090-566e-4079-895a-f9f1e99bb686" containerName="authorino"
Jul 09 23:40:01.672529 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.672506    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-849b8469f6-m5d4c"
Jul 09 23:40:01.675885 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.675868    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 09 23:40:01.675993 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.675868    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 09 23:40:01.675993 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.675868    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kbmvl\""
Jul 09 23:40:01.683008 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.682469    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-849b8469f6-m5d4c"]
Jul 09 23:40:01.770078 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.770030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b80460ea-9c86-43c1-a59e-97c841313545-webhook-cert\") pod \"maas-controller-849b8469f6-m5d4c\" (UID: \"b80460ea-9c86-43c1-a59e-97c841313545\") " pod="opendatahub/maas-controller-849b8469f6-m5d4c"
Jul 09 23:40:01.770078 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.770081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnb9z\" (UniqueName: \"kubernetes.io/projected/b80460ea-9c86-43c1-a59e-97c841313545-kube-api-access-gnb9z\") pod \"maas-controller-849b8469f6-m5d4c\" (UID: \"b80460ea-9c86-43c1-a59e-97c841313545\") " pod="opendatahub/maas-controller-849b8469f6-m5d4c"
Jul 09 23:40:01.871251 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.871207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b80460ea-9c86-43c1-a59e-97c841313545-webhook-cert\") pod \"maas-controller-849b8469f6-m5d4c\" (UID: \"b80460ea-9c86-43c1-a59e-97c841313545\") " pod="opendatahub/maas-controller-849b8469f6-m5d4c"
Jul 09 23:40:01.871251 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.871251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnb9z\" (UniqueName: \"kubernetes.io/projected/b80460ea-9c86-43c1-a59e-97c841313545-kube-api-access-gnb9z\") pod \"maas-controller-849b8469f6-m5d4c\" (UID: \"b80460ea-9c86-43c1-a59e-97c841313545\") " pod="opendatahub/maas-controller-849b8469f6-m5d4c"
Jul 09 23:40:01.873737 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.873708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b80460ea-9c86-43c1-a59e-97c841313545-webhook-cert\") pod \"maas-controller-849b8469f6-m5d4c\" (UID: \"b80460ea-9c86-43c1-a59e-97c841313545\") " pod="opendatahub/maas-controller-849b8469f6-m5d4c"
Jul 09 23:40:01.878352 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.878325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnb9z\" (UniqueName: \"kubernetes.io/projected/b80460ea-9c86-43c1-a59e-97c841313545-kube-api-access-gnb9z\") pod \"maas-controller-849b8469f6-m5d4c\" (UID: \"b80460ea-9c86-43c1-a59e-97c841313545\") " pod="opendatahub/maas-controller-849b8469f6-m5d4c"
Jul 09 23:40:01.986142 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:01.986047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-849b8469f6-m5d4c"
Jul 09 23:40:02.121701 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:02.121647    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-849b8469f6-m5d4c"]
Jul 09 23:40:02.419270 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:02.419228    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efffe372-198b-4aaf-bb86-55f33ea647f6" path="/var/lib/kubelet/pods/efffe372-198b-4aaf-bb86-55f33ea647f6/volumes"
Jul 09 23:40:02.790836 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:02.790813    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 09 23:40:02.896810 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:02.896772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-849b8469f6-m5d4c" event={"ID":"b80460ea-9c86-43c1-a59e-97c841313545","Type":"ContainerStarted","Data":"09023d943cad8f0f377a0a23aeb7e6eabd7a565b8799a461be8a4e335a861384"}
Jul 09 23:40:02.896984 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:02.896836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-849b8469f6-m5d4c" event={"ID":"b80460ea-9c86-43c1-a59e-97c841313545","Type":"ContainerStarted","Data":"28fab7d13e148b808da5a6e81a94aedff457313a4b783bbf8617b023439131d1"}
Jul 09 23:40:02.896984 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:02.896879    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-849b8469f6-m5d4c"
Jul 09 23:40:02.912265 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:02.912211    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-849b8469f6-m5d4c" podStartSLOduration=1.250804035 podStartE2EDuration="1.912194032s" totalImagesPullingTime="661.389997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:40:02.911174294 +0000 UTC m=+721.055105080" watchObservedRunningTime="2026-07-09 23:40:02.912194032 +0000 UTC m=+721.056124864"
Jul 09 23:40:13.906357 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:40:13.906316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-849b8469f6-m5d4c"
Jul 09 23:41:00.746156 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:00.746046    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc" podUID="686df5b5-af27-4125-8c70-5b8f0a52afdf" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/readyz\": dial tcp 10.134.0.23:8081: connect: connection refused"
Jul 09 23:41:01.199499 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:01.199460    2580 generic.go:356] "Generic (PLEG): container finished" podID="686df5b5-af27-4125-8c70-5b8f0a52afdf" containerID="65cf6163eedcc3c3c81c41343bb2cbf1edf84bef5cba736e6bc3e7734ba9268e" exitCode=1
Jul 09 23:41:01.199700 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:01.199531    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc" event={"ID":"686df5b5-af27-4125-8c70-5b8f0a52afdf","Type":"ContainerDied","Data":"65cf6163eedcc3c3c81c41343bb2cbf1edf84bef5cba736e6bc3e7734ba9268e"}
Jul 09 23:41:01.199917 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:01.199903    2580 scope.go:117] "RemoveContainer" containerID="65cf6163eedcc3c3c81c41343bb2cbf1edf84bef5cba736e6bc3e7734ba9268e"
Jul 09 23:41:02.206666 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:02.206624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc" event={"ID":"686df5b5-af27-4125-8c70-5b8f0a52afdf","Type":"ContainerStarted","Data":"89b50790fe9a75ee8eba4ca9e6f8effe45dd1c5ef1a8b4981e4bac791014a1f1"}
Jul 09 23:41:02.207151 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:02.206910    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:41:12.263215 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:12.263185    2580 generic.go:356] "Generic (PLEG): container finished" podID="686df5b5-af27-4125-8c70-5b8f0a52afdf" containerID="89b50790fe9a75ee8eba4ca9e6f8effe45dd1c5ef1a8b4981e4bac791014a1f1" exitCode=1
Jul 09 23:41:12.263625 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:12.263251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc" event={"ID":"686df5b5-af27-4125-8c70-5b8f0a52afdf","Type":"ContainerDied","Data":"89b50790fe9a75ee8eba4ca9e6f8effe45dd1c5ef1a8b4981e4bac791014a1f1"}
Jul 09 23:41:12.263625 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:12.263301    2580 scope.go:117] "RemoveContainer" containerID="65cf6163eedcc3c3c81c41343bb2cbf1edf84bef5cba736e6bc3e7734ba9268e"
Jul 09 23:41:12.263759 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:12.263742    2580 scope.go:117] "RemoveContainer" containerID="89b50790fe9a75ee8eba4ca9e6f8effe45dd1c5ef1a8b4981e4bac791014a1f1"
Jul 09 23:41:12.263970 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:41:12.263947    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-l9vqc_opendatahub(686df5b5-af27-4125-8c70-5b8f0a52afdf)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc" podUID="686df5b5-af27-4125-8c70-5b8f0a52afdf"
Jul 09 23:41:15.523485 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:15.523441    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:41:15.524011 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:15.523915    2580 scope.go:117] "RemoveContainer" containerID="89b50790fe9a75ee8eba4ca9e6f8effe45dd1c5ef1a8b4981e4bac791014a1f1"
Jul 09 23:41:15.524177 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:41:15.524152    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-l9vqc_opendatahub(686df5b5-af27-4125-8c70-5b8f0a52afdf)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc" podUID="686df5b5-af27-4125-8c70-5b8f0a52afdf"
Jul 09 23:41:29.413936 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:29.413902    2580 scope.go:117] "RemoveContainer" containerID="89b50790fe9a75ee8eba4ca9e6f8effe45dd1c5ef1a8b4981e4bac791014a1f1"
Jul 09 23:41:30.355826 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:30.355793    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc" event={"ID":"686df5b5-af27-4125-8c70-5b8f0a52afdf","Type":"ContainerStarted","Data":"4c0f37a659846c404586b84c7280d403c0dda82bd2d397094d00ad2577e7949d"}
Jul 09 23:41:30.356024 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:41:30.356004    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:42:01.367382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:42:01.367348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l9vqc"
Jul 09 23:45:00.140872 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:00.140834    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g"]
Jul 09 23:45:00.144017 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:00.143992    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g"
Jul 09 23:45:00.146468 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:00.146444    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5vt95\""
Jul 09 23:45:00.154592 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:00.154571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g"
Jul 09 23:45:00.158163 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:00.158139    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g"]
Jul 09 23:45:00.302517 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:00.302492    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g"]
Jul 09 23:45:00.305059 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:45:00.305028    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6fd61a_898f_434b_b1b9_60ee2931ab63.slice/crio-07b5eca6bf813604bd4ece45264182ec68577159ca3dd80b256187caf63cb52f WatchSource:0}: Error finding container 07b5eca6bf813604bd4ece45264182ec68577159ca3dd80b256187caf63cb52f: Status 404 returned error can't find the container with id 07b5eca6bf813604bd4ece45264182ec68577159ca3dd80b256187caf63cb52f
Jul 09 23:45:00.307568 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:00.307551    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 23:45:00.445756 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:00.445641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g" event={"ID":"fa6fd61a-898f-434b-b1b9-60ee2931ab63","Type":"ContainerStarted","Data":"07b5eca6bf813604bd4ece45264182ec68577159ca3dd80b256187caf63cb52f"}
Jul 09 23:45:00.786014 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:00.785980    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"]
Jul 09 23:45:00.786208 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:00.786162    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj" podUID="ed06cb77-5234-4427-ba27-c0e7a0026b02" containerName="manager" containerID="cri-o://05b56e0a8c1b7f189b320896e0ff27ca3bac54bdb7182a08d91a75948562e1ed" gracePeriod=10
Jul 09 23:45:03.651568 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:03.651541    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"
Jul 09 23:45:03.768193 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:03.768163    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed06cb77-5234-4427-ba27-c0e7a0026b02-extensions-socket-volume\") pod \"ed06cb77-5234-4427-ba27-c0e7a0026b02\" (UID: \"ed06cb77-5234-4427-ba27-c0e7a0026b02\") "
Jul 09 23:45:03.768353 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:03.768217    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6ssv\" (UniqueName: \"kubernetes.io/projected/ed06cb77-5234-4427-ba27-c0e7a0026b02-kube-api-access-x6ssv\") pod \"ed06cb77-5234-4427-ba27-c0e7a0026b02\" (UID: \"ed06cb77-5234-4427-ba27-c0e7a0026b02\") "
Jul 09 23:45:03.768557 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:03.768534    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed06cb77-5234-4427-ba27-c0e7a0026b02-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ed06cb77-5234-4427-ba27-c0e7a0026b02" (UID: "ed06cb77-5234-4427-ba27-c0e7a0026b02"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:45:03.770269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:03.770245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed06cb77-5234-4427-ba27-c0e7a0026b02-kube-api-access-x6ssv" (OuterVolumeSpecName: "kube-api-access-x6ssv") pod "ed06cb77-5234-4427-ba27-c0e7a0026b02" (UID: "ed06cb77-5234-4427-ba27-c0e7a0026b02"). InnerVolumeSpecName "kube-api-access-x6ssv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:45:03.869084 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:03.869048    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed06cb77-5234-4427-ba27-c0e7a0026b02-extensions-socket-volume\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:45:03.869084 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:03.869077    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6ssv\" (UniqueName: \"kubernetes.io/projected/ed06cb77-5234-4427-ba27-c0e7a0026b02-kube-api-access-x6ssv\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:45:04.468267 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:04.468230    2580 generic.go:356] "Generic (PLEG): container finished" podID="ed06cb77-5234-4427-ba27-c0e7a0026b02" containerID="05b56e0a8c1b7f189b320896e0ff27ca3bac54bdb7182a08d91a75948562e1ed" exitCode=0
Jul 09 23:45:04.468426 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:04.468276    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"
Jul 09 23:45:04.468426 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:04.468297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj" event={"ID":"ed06cb77-5234-4427-ba27-c0e7a0026b02","Type":"ContainerDied","Data":"05b56e0a8c1b7f189b320896e0ff27ca3bac54bdb7182a08d91a75948562e1ed"}
Jul 09 23:45:04.468426 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:04.468350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj" event={"ID":"ed06cb77-5234-4427-ba27-c0e7a0026b02","Type":"ContainerDied","Data":"7efa252a67af3f9735efdc335f974193d93915863dcf9244edd1d044feb9ede4"}
Jul 09 23:45:04.468426 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:04.468383    2580 scope.go:117] "RemoveContainer" containerID="05b56e0a8c1b7f189b320896e0ff27ca3bac54bdb7182a08d91a75948562e1ed"
Jul 09 23:45:04.478133 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:04.478112    2580 scope.go:117] "RemoveContainer" containerID="05b56e0a8c1b7f189b320896e0ff27ca3bac54bdb7182a08d91a75948562e1ed"
Jul 09 23:45:04.478416 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:45:04.478396    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b56e0a8c1b7f189b320896e0ff27ca3bac54bdb7182a08d91a75948562e1ed\": container with ID starting with 05b56e0a8c1b7f189b320896e0ff27ca3bac54bdb7182a08d91a75948562e1ed not found: ID does not exist" containerID="05b56e0a8c1b7f189b320896e0ff27ca3bac54bdb7182a08d91a75948562e1ed"
Jul 09 23:45:04.478486 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:04.478428    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b56e0a8c1b7f189b320896e0ff27ca3bac54bdb7182a08d91a75948562e1ed"} err="failed to get container status \"05b56e0a8c1b7f189b320896e0ff27ca3bac54bdb7182a08d91a75948562e1ed\": rpc error: code = NotFound desc = could not find container \"05b56e0a8c1b7f189b320896e0ff27ca3bac54bdb7182a08d91a75948562e1ed\": container with ID starting with 05b56e0a8c1b7f189b320896e0ff27ca3bac54bdb7182a08d91a75948562e1ed not found: ID does not exist"
Jul 09 23:45:04.487000 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:04.486976    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"]
Jul 09 23:45:04.490467 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:04.490440    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hzhj"]
Jul 09 23:45:06.419699 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:06.419648    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed06cb77-5234-4427-ba27-c0e7a0026b02" path="/var/lib/kubelet/pods/ed06cb77-5234-4427-ba27-c0e7a0026b02/volumes"
Jul 09 23:45:08.493032 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:08.492995    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g" event={"ID":"fa6fd61a-898f-434b-b1b9-60ee2931ab63","Type":"ContainerStarted","Data":"ff4fdd3cbc83e19da46bf18d2922df0fb4e08b02c584b1412e98be9e97a3c7e2"}
Jul 09 23:45:08.506875 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:45:08.506824    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g" podStartSLOduration=1.184635575 podStartE2EDuration="8.50680767s" totalImagesPullingTime="7.322172095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:45:08.505950905 +0000 UTC m=+1026.649881692" watchObservedRunningTime="2026-07-09 23:45:08.50680767 +0000 UTC m=+1026.650738458"
Jul 09 23:46:00.009368 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:00.009285    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g"]
Jul 09 23:46:00.009899 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:00.009490    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g" podUID="fa6fd61a-898f-434b-b1b9-60ee2931ab63" containerName="cleanup" containerID="cri-o://ff4fdd3cbc83e19da46bf18d2922df0fb4e08b02c584b1412e98be9e97a3c7e2" gracePeriod=30
Jul 09 23:46:01.908045 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:01.908007    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs"]
Jul 09 23:46:01.908463 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:01.908383    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed06cb77-5234-4427-ba27-c0e7a0026b02" containerName="manager"
Jul 09 23:46:01.908463 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:01.908398    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed06cb77-5234-4427-ba27-c0e7a0026b02" containerName="manager"
Jul 09 23:46:01.908634 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:01.908505    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed06cb77-5234-4427-ba27-c0e7a0026b02" containerName="manager"
Jul 09 23:46:01.911845 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:01.911823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs"
Jul 09 23:46:01.914288 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:01.914265    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9tb6j\""
Jul 09 23:46:01.923031 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:01.922999    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs"]
Jul 09 23:46:01.955633 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:01.955598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788pd\" (UniqueName: \"kubernetes.io/projected/363555ca-801c-4d8f-b5f0-4600041c5a2a-kube-api-access-788pd\") pod \"kuadrant-operator-controller-manager-86878c769f-k6zgs\" (UID: \"363555ca-801c-4d8f-b5f0-4600041c5a2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs"
Jul 09 23:46:01.955633 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:01.955634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/363555ca-801c-4d8f-b5f0-4600041c5a2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k6zgs\" (UID: \"363555ca-801c-4d8f-b5f0-4600041c5a2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs"
Jul 09 23:46:02.056906 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:02.056870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788pd\" (UniqueName: \"kubernetes.io/projected/363555ca-801c-4d8f-b5f0-4600041c5a2a-kube-api-access-788pd\") pod \"kuadrant-operator-controller-manager-86878c769f-k6zgs\" (UID: \"363555ca-801c-4d8f-b5f0-4600041c5a2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs"
Jul 09 23:46:02.056906 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:02.056908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/363555ca-801c-4d8f-b5f0-4600041c5a2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k6zgs\" (UID: \"363555ca-801c-4d8f-b5f0-4600041c5a2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs"
Jul 09 23:46:02.057259 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:02.057243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/363555ca-801c-4d8f-b5f0-4600041c5a2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k6zgs\" (UID: \"363555ca-801c-4d8f-b5f0-4600041c5a2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs"
Jul 09 23:46:02.064714 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:02.064688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-788pd\" (UniqueName: \"kubernetes.io/projected/363555ca-801c-4d8f-b5f0-4600041c5a2a-kube-api-access-788pd\") pod \"kuadrant-operator-controller-manager-86878c769f-k6zgs\" (UID: \"363555ca-801c-4d8f-b5f0-4600041c5a2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs"
Jul 09 23:46:02.222589 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:02.222498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs"
Jul 09 23:46:02.374564 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:02.374533    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs"]
Jul 09 23:46:02.379864 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:46:02.379833    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363555ca_801c_4d8f_b5f0_4600041c5a2a.slice/crio-8b75aca45168021ee787bdffca2491de7f2f0699070f92c1391e61b68a9ea5f3 WatchSource:0}: Error finding container 8b75aca45168021ee787bdffca2491de7f2f0699070f92c1391e61b68a9ea5f3: Status 404 returned error can't find the container with id 8b75aca45168021ee787bdffca2491de7f2f0699070f92c1391e61b68a9ea5f3
Jul 09 23:46:02.762178 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:02.762137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs" event={"ID":"363555ca-801c-4d8f-b5f0-4600041c5a2a","Type":"ContainerStarted","Data":"d0d3a16671dd904ef449e42e1042a0253aade978c9bee283020dd2a7233db568"}
Jul 09 23:46:02.762384 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:02.762198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs" event={"ID":"363555ca-801c-4d8f-b5f0-4600041c5a2a","Type":"ContainerStarted","Data":"8b75aca45168021ee787bdffca2491de7f2f0699070f92c1391e61b68a9ea5f3"}
Jul 09 23:46:02.762384 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:02.762223    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs"
Jul 09 23:46:02.781685 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:02.781611    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs" podStartSLOduration=1.781597343 podStartE2EDuration="1.781597343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:46:02.780635348 +0000 UTC m=+1080.924566135" watchObservedRunningTime="2026-07-09 23:46:02.781597343 +0000 UTC m=+1080.925528128"
Jul 09 23:46:13.769418 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:13.769383    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k6zgs"
Jul 09 23:46:30.194027 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:30.193999    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g"
Jul 09 23:46:30.910141 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:30.910109    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa6fd61a-898f-434b-b1b9-60ee2931ab63" containerID="ff4fdd3cbc83e19da46bf18d2922df0fb4e08b02c584b1412e98be9e97a3c7e2" exitCode=137
Jul 09 23:46:30.910404 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:30.910148    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g"
Jul 09 23:46:30.910404 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:30.910202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g" event={"ID":"fa6fd61a-898f-434b-b1b9-60ee2931ab63","Type":"ContainerDied","Data":"ff4fdd3cbc83e19da46bf18d2922df0fb4e08b02c584b1412e98be9e97a3c7e2"}
Jul 09 23:46:30.910404 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:30.910254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g" event={"ID":"fa6fd61a-898f-434b-b1b9-60ee2931ab63","Type":"ContainerDied","Data":"07b5eca6bf813604bd4ece45264182ec68577159ca3dd80b256187caf63cb52f"}
Jul 09 23:46:30.910404 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:30.910276    2580 scope.go:117] "RemoveContainer" containerID="ff4fdd3cbc83e19da46bf18d2922df0fb4e08b02c584b1412e98be9e97a3c7e2"
Jul 09 23:46:30.918801 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:30.918782    2580 scope.go:117] "RemoveContainer" containerID="ff4fdd3cbc83e19da46bf18d2922df0fb4e08b02c584b1412e98be9e97a3c7e2"
Jul 09 23:46:30.919072 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:46:30.919052    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4fdd3cbc83e19da46bf18d2922df0fb4e08b02c584b1412e98be9e97a3c7e2\": container with ID starting with ff4fdd3cbc83e19da46bf18d2922df0fb4e08b02c584b1412e98be9e97a3c7e2 not found: ID does not exist" containerID="ff4fdd3cbc83e19da46bf18d2922df0fb4e08b02c584b1412e98be9e97a3c7e2"
Jul 09 23:46:30.919115 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:30.919085    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4fdd3cbc83e19da46bf18d2922df0fb4e08b02c584b1412e98be9e97a3c7e2"} err="failed to get container status \"ff4fdd3cbc83e19da46bf18d2922df0fb4e08b02c584b1412e98be9e97a3c7e2\": rpc error: code = NotFound desc = could not find container \"ff4fdd3cbc83e19da46bf18d2922df0fb4e08b02c584b1412e98be9e97a3c7e2\": container with ID starting with ff4fdd3cbc83e19da46bf18d2922df0fb4e08b02c584b1412e98be9e97a3c7e2 not found: ID does not exist"
Jul 09 23:46:30.927117 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:30.927091    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g"]
Jul 09 23:46:30.929854 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:30.929830    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727345-pqm9g"]
Jul 09 23:46:32.419172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:46:32.419136    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa6fd61a-898f-434b-b1b9-60ee2931ab63" path="/var/lib/kubelet/pods/fa6fd61a-898f-434b-b1b9-60ee2931ab63/volumes"
Jul 09 23:52:23.095069 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.094975    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"]
Jul 09 23:52:23.097477 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.095397    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6fd61a-898f-434b-b1b9-60ee2931ab63" containerName="cleanup"
Jul 09 23:52:23.097477 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.095412    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6fd61a-898f-434b-b1b9-60ee2931ab63" containerName="cleanup"
Jul 09 23:52:23.097477 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.095483    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa6fd61a-898f-434b-b1b9-60ee2931ab63" containerName="cleanup"
Jul 09 23:52:23.098405 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.098384    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.101222 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.101199    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d2fc08b2-openshift-default-dockercfg-kbnf8\""
Jul 09 23:52:23.101335 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.101223    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 23:52:23.122059 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.122027    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"]
Jul 09 23:52:23.172638 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.172604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-workload-socket\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.172825 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.172717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghbd9\" (UniqueName: \"kubernetes.io/projected/d23df273-acef-4823-9eb8-75bab1dfd75e-kube-api-access-ghbd9\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.172825 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.172760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-token\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.172825 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.172819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-workload-certs\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.172966 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.172883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-envoy\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.172966 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.172905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23df273-acef-4823-9eb8-75bab1dfd75e-istiod-ca-cert\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.173080 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.173006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-data\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.173080 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.173042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-credential-socket\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.173080 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.173072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-podinfo\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.274220 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.274183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-podinfo\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.274418 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.274230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-workload-socket\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.274418 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.274256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghbd9\" (UniqueName: \"kubernetes.io/projected/d23df273-acef-4823-9eb8-75bab1dfd75e-kube-api-access-ghbd9\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.274418 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.274284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-token\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.274418 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.274326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-workload-certs\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.274418 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.274360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-envoy\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.274418 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.274383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23df273-acef-4823-9eb8-75bab1dfd75e-istiod-ca-cert\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.274881 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.274426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-data\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.274881 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.274450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-credential-socket\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.274881 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.274674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-workload-socket\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.274881 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.274772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-workload-certs\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.274881 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.274835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-data\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.274881 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.274837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-credential-socket\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.275272 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.275240    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23df273-acef-4823-9eb8-75bab1dfd75e-istiod-ca-cert\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.276546 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.276525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-envoy\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.277199 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.277173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-podinfo\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.281888 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.281864    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-token\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.282083 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.282069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghbd9\" (UniqueName: \"kubernetes.io/projected/d23df273-acef-4823-9eb8-75bab1dfd75e-kube-api-access-ghbd9\") pod \"e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") " pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.409630 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.409536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:23.547032 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.547004    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"]
Jul 09 23:52:23.548141 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:52:23.548112    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23df273_acef_4823_9eb8_75bab1dfd75e.slice/crio-82bba5b9db6cde60a0c898abb78e2ba2dad418529d8f160f3d7922396dabdb54 WatchSource:0}: Error finding container 82bba5b9db6cde60a0c898abb78e2ba2dad418529d8f160f3d7922396dabdb54: Status 404 returned error can't find the container with id 82bba5b9db6cde60a0c898abb78e2ba2dad418529d8f160f3d7922396dabdb54
Jul 09 23:52:23.550255 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.550242    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 23:52:23.756922 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:23.756832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl" event={"ID":"d23df273-acef-4823-9eb8-75bab1dfd75e","Type":"ContainerStarted","Data":"82bba5b9db6cde60a0c898abb78e2ba2dad418529d8f160f3d7922396dabdb54"}
Jul 09 23:52:24.397264 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.397229    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"]
Jul 09 23:52:24.401823 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.401803    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:24.404099 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.404080    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d2fc08b2-serving-cert\""
Jul 09 23:52:24.411039 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.411013    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"]
Jul 09 23:52:24.484348 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.484305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-openshift-service-ca\") pod \"maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz\" (UID: \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:24.484557 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.484438    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5mq\" (UniqueName: \"kubernetes.io/projected/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-kube-api-access-cb5mq\") pod \"maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz\" (UID: \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:24.484557 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.484474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-maas-api-tls\") pod \"maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz\" (UID: \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:24.585088 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.585054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-openshift-service-ca\") pod \"maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz\" (UID: \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:24.585285 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.585161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5mq\" (UniqueName: \"kubernetes.io/projected/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-kube-api-access-cb5mq\") pod \"maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz\" (UID: \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:24.585285 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.585190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-maas-api-tls\") pod \"maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz\" (UID: \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:24.585787 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.585760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-openshift-service-ca\") pod \"maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz\" (UID: \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:24.587666 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.587610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-maas-api-tls\") pod \"maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz\" (UID: \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:24.593365 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.593341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5mq\" (UniqueName: \"kubernetes.io/projected/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-kube-api-access-cb5mq\") pod \"maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz\" (UID: \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:24.715632 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.715543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:24.866574 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:24.866551    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"]
Jul 09 23:52:24.868882 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:52:24.868841    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28998cba_3622_41b9_9c86_2ce6f2ac9bb5.slice/crio-6a76a42e3d12dc19db7b52315af98bc2d1b445d102b28b0b8235e6dfc3b079d3 WatchSource:0}: Error finding container 6a76a42e3d12dc19db7b52315af98bc2d1b445d102b28b0b8235e6dfc3b079d3: Status 404 returned error can't find the container with id 6a76a42e3d12dc19db7b52315af98bc2d1b445d102b28b0b8235e6dfc3b079d3
Jul 09 23:52:25.770092 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:25.770052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz" event={"ID":"28998cba-3622-41b9-9c86-2ce6f2ac9bb5","Type":"ContainerStarted","Data":"dda2ccb798b71ac8b30e7eddaee5a93b36d054706027bd9b43a7f3173ddf5d26"}
Jul 09 23:52:25.770567 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:25.770107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz" event={"ID":"28998cba-3622-41b9-9c86-2ce6f2ac9bb5","Type":"ContainerStarted","Data":"6a76a42e3d12dc19db7b52315af98bc2d1b445d102b28b0b8235e6dfc3b079d3"}
Jul 09 23:52:25.770567 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:25.770180    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:25.788993 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:25.788889    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz" podStartSLOduration=1.134826186 podStartE2EDuration="1.788873598s" totalImagesPullingTime="654.047412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:52:25.786433266 +0000 UTC m=+1463.930364058" watchObservedRunningTime="2026-07-09 23:52:25.788873598 +0000 UTC m=+1463.932804383"
Jul 09 23:52:26.959562 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:26.959527    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:52:26.959848 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:26.959597    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:52:26.959848 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:26.959627    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:52:27.781786 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:27.781752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl" event={"ID":"d23df273-acef-4823-9eb8-75bab1dfd75e","Type":"ContainerStarted","Data":"808115ac89d52a6c184dba52ae88776f8536301fd0951e43cd6ce1ae4691e180"}
Jul 09 23:52:27.801103 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:27.801047    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl" podStartSLOduration=1.392153902 podStartE2EDuration="4.801030443s" totalImagesPullingTime="3.408876541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:52:27.799882954 +0000 UTC m=+1465.943813732" watchObservedRunningTime="2026-07-09 23:52:27.801030443 +0000 UTC m=+1465.944961230"
Jul 09 23:52:28.410620 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:28.410578    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:28.419681 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:28.419636    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:28.787492 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:28.787465    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:28.788573 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:28.788551    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:29.466228 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:29.466188    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"]
Jul 09 23:52:29.638567 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:29.638534    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"]
Jul 09 23:52:29.638798 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:29.638731    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz" podUID="28998cba-3622-41b9-9c86-2ce6f2ac9bb5" containerName="maas-api" containerID="cri-o://dda2ccb798b71ac8b30e7eddaee5a93b36d054706027bd9b43a7f3173ddf5d26" gracePeriod=30
Jul 09 23:52:29.794451 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:29.794419    2580 generic.go:356] "Generic (PLEG): container finished" podID="28998cba-3622-41b9-9c86-2ce6f2ac9bb5" containerID="dda2ccb798b71ac8b30e7eddaee5a93b36d054706027bd9b43a7f3173ddf5d26" exitCode=0
Jul 09 23:52:29.794613 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:29.794494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz" event={"ID":"28998cba-3622-41b9-9c86-2ce6f2ac9bb5","Type":"ContainerDied","Data":"dda2ccb798b71ac8b30e7eddaee5a93b36d054706027bd9b43a7f3173ddf5d26"}
Jul 09 23:52:29.794796 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:29.794781    2580 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl" secret="" err="secret \"e2e-ait-d2fc08b2-openshift-default-dockercfg-kbnf8\" not found"
Jul 09 23:52:29.891338 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:29.891301    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:30.035372 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.035339    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-openshift-service-ca\") pod \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\" (UID: \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\") "
Jul 09 23:52:30.035557 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.035386    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb5mq\" (UniqueName: \"kubernetes.io/projected/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-kube-api-access-cb5mq\") pod \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\" (UID: \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\") "
Jul 09 23:52:30.035557 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.035411    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-maas-api-tls\") pod \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\" (UID: \"28998cba-3622-41b9-9c86-2ce6f2ac9bb5\") "
Jul 09 23:52:30.035850 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.035822    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "28998cba-3622-41b9-9c86-2ce6f2ac9bb5" (UID: "28998cba-3622-41b9-9c86-2ce6f2ac9bb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 23:52:30.037602 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.037578    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "28998cba-3622-41b9-9c86-2ce6f2ac9bb5" (UID: "28998cba-3622-41b9-9c86-2ce6f2ac9bb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 23:52:30.037815 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.037638    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-kube-api-access-cb5mq" (OuterVolumeSpecName: "kube-api-access-cb5mq") pod "28998cba-3622-41b9-9c86-2ce6f2ac9bb5" (UID: "28998cba-3622-41b9-9c86-2ce6f2ac9bb5"). InnerVolumeSpecName "kube-api-access-cb5mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:52:30.136974 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.136932    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-openshift-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:30.136974 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.136966    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb5mq\" (UniqueName: \"kubernetes.io/projected/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-kube-api-access-cb5mq\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:30.136974 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.136978    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28998cba-3622-41b9-9c86-2ce6f2ac9bb5-maas-api-tls\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:30.800535 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.800496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz" event={"ID":"28998cba-3622-41b9-9c86-2ce6f2ac9bb5","Type":"ContainerDied","Data":"6a76a42e3d12dc19db7b52315af98bc2d1b445d102b28b0b8235e6dfc3b079d3"}
Jul 09 23:52:30.800535 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.800531    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"
Jul 09 23:52:30.801049 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.800570    2580 scope.go:117] "RemoveContainer" containerID="dda2ccb798b71ac8b30e7eddaee5a93b36d054706027bd9b43a7f3173ddf5d26"
Jul 09 23:52:30.801049 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.800596    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl" podUID="d23df273-acef-4823-9eb8-75bab1dfd75e" containerName="istio-proxy" containerID="cri-o://808115ac89d52a6c184dba52ae88776f8536301fd0951e43cd6ce1ae4691e180" gracePeriod=30
Jul 09 23:52:30.831567 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.831530    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"]
Jul 09 23:52:30.836402 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:30.836360    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d2fc08b2-8794d8969-tbddz"]
Jul 09 23:52:32.420062 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:32.420027    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28998cba-3622-41b9-9c86-2ce6f2ac9bb5" path="/var/lib/kubelet/pods/28998cba-3622-41b9-9c86-2ce6f2ac9bb5/volumes"
Jul 09 23:52:36.064860 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.064795    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:36.089158 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.089125    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-token\") pod \"d23df273-acef-4823-9eb8-75bab1dfd75e\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") "
Jul 09 23:52:36.089342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.089197    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-envoy\") pod \"d23df273-acef-4823-9eb8-75bab1dfd75e\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") "
Jul 09 23:52:36.089342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.089224    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-credential-socket\") pod \"d23df273-acef-4823-9eb8-75bab1dfd75e\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") "
Jul 09 23:52:36.089342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.089273    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghbd9\" (UniqueName: \"kubernetes.io/projected/d23df273-acef-4823-9eb8-75bab1dfd75e-kube-api-access-ghbd9\") pod \"d23df273-acef-4823-9eb8-75bab1dfd75e\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") "
Jul 09 23:52:36.089342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.089304    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-workload-certs\") pod \"d23df273-acef-4823-9eb8-75bab1dfd75e\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") "
Jul 09 23:52:36.089342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.089332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-podinfo\") pod \"d23df273-acef-4823-9eb8-75bab1dfd75e\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") "
Jul 09 23:52:36.089594 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.089372    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-data\") pod \"d23df273-acef-4823-9eb8-75bab1dfd75e\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") "
Jul 09 23:52:36.089594 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.089417    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-workload-socket\") pod \"d23df273-acef-4823-9eb8-75bab1dfd75e\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") "
Jul 09 23:52:36.089594 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.089444    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23df273-acef-4823-9eb8-75bab1dfd75e-istiod-ca-cert\") pod \"d23df273-acef-4823-9eb8-75bab1dfd75e\" (UID: \"d23df273-acef-4823-9eb8-75bab1dfd75e\") "
Jul 09 23:52:36.089941 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.089731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d23df273-acef-4823-9eb8-75bab1dfd75e" (UID: "d23df273-acef-4823-9eb8-75bab1dfd75e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:52:36.089941 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.089864    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d23df273-acef-4823-9eb8-75bab1dfd75e" (UID: "d23df273-acef-4823-9eb8-75bab1dfd75e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:52:36.090480 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.090162    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23df273-acef-4823-9eb8-75bab1dfd75e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d23df273-acef-4823-9eb8-75bab1dfd75e" (UID: "d23df273-acef-4823-9eb8-75bab1dfd75e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 23:52:36.090480 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.090351    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d23df273-acef-4823-9eb8-75bab1dfd75e" (UID: "d23df273-acef-4823-9eb8-75bab1dfd75e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:52:36.090480 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.090385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-data" (OuterVolumeSpecName: "istio-data") pod "d23df273-acef-4823-9eb8-75bab1dfd75e" (UID: "d23df273-acef-4823-9eb8-75bab1dfd75e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:52:36.091801 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.091764    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-token" (OuterVolumeSpecName: "istio-token") pod "d23df273-acef-4823-9eb8-75bab1dfd75e" (UID: "d23df273-acef-4823-9eb8-75bab1dfd75e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:52:36.092152 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.092131    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d23df273-acef-4823-9eb8-75bab1dfd75e" (UID: "d23df273-acef-4823-9eb8-75bab1dfd75e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:52:36.092259 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.092235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23df273-acef-4823-9eb8-75bab1dfd75e-kube-api-access-ghbd9" (OuterVolumeSpecName: "kube-api-access-ghbd9") pod "d23df273-acef-4823-9eb8-75bab1dfd75e" (UID: "d23df273-acef-4823-9eb8-75bab1dfd75e"). InnerVolumeSpecName "kube-api-access-ghbd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:52:36.092470 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.092444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d23df273-acef-4823-9eb8-75bab1dfd75e" (UID: "d23df273-acef-4823-9eb8-75bab1dfd75e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 23:52:36.190847 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.190807    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-credential-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:36.190847 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.190836    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghbd9\" (UniqueName: \"kubernetes.io/projected/d23df273-acef-4823-9eb8-75bab1dfd75e-kube-api-access-ghbd9\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:36.190847 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.190849    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-workload-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:36.190847 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.190858    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-podinfo\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:36.191110 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.190867    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-data\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:36.191110 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.190875    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-workload-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:36.191110 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.190882    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23df273-acef-4823-9eb8-75bab1dfd75e-istiod-ca-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:36.191110 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.190889    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-token\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:36.191110 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.190896    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23df273-acef-4823-9eb8-75bab1dfd75e-istio-envoy\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:36.683045 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.682941    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"]
Jul 09 23:52:36.683507 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.683485    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28998cba-3622-41b9-9c86-2ce6f2ac9bb5" containerName="maas-api"
Jul 09 23:52:36.683507 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.683505    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28998cba-3622-41b9-9c86-2ce6f2ac9bb5" containerName="maas-api"
Jul 09 23:52:36.683779 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.683519    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23df273-acef-4823-9eb8-75bab1dfd75e" containerName="istio-proxy"
Jul 09 23:52:36.683779 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.683529    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23df273-acef-4823-9eb8-75bab1dfd75e" containerName="istio-proxy"
Jul 09 23:52:36.683779 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.683636    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23df273-acef-4823-9eb8-75bab1dfd75e" containerName="istio-proxy"
Jul 09 23:52:36.683779 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.683669    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28998cba-3622-41b9-9c86-2ce6f2ac9bb5" containerName="maas-api"
Jul 09 23:52:36.687347 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.687321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:36.689622 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.689602    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-143fdde2-serving-cert\""
Jul 09 23:52:36.694549 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.694505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-maas-api-tls\") pod \"maas-api-e2e-ait-143fdde2-7b46869895-5x84x\" (UID: \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:36.694690 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.694593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-openshift-service-ca\") pod \"maas-api-e2e-ait-143fdde2-7b46869895-5x84x\" (UID: \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:36.694690 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.694634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j8sr\" (UniqueName: \"kubernetes.io/projected/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-kube-api-access-7j8sr\") pod \"maas-api-e2e-ait-143fdde2-7b46869895-5x84x\" (UID: \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:36.698454 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.698324    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"]
Jul 09 23:52:36.795741 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.795698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-maas-api-tls\") pod \"maas-api-e2e-ait-143fdde2-7b46869895-5x84x\" (UID: \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:36.795943 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.795760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-openshift-service-ca\") pod \"maas-api-e2e-ait-143fdde2-7b46869895-5x84x\" (UID: \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:36.795943 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.795793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j8sr\" (UniqueName: \"kubernetes.io/projected/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-kube-api-access-7j8sr\") pod \"maas-api-e2e-ait-143fdde2-7b46869895-5x84x\" (UID: \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:36.796490 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.796461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-openshift-service-ca\") pod \"maas-api-e2e-ait-143fdde2-7b46869895-5x84x\" (UID: \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:36.798153 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.798135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-maas-api-tls\") pod \"maas-api-e2e-ait-143fdde2-7b46869895-5x84x\" (UID: \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:36.810378 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.810355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j8sr\" (UniqueName: \"kubernetes.io/projected/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-kube-api-access-7j8sr\") pod \"maas-api-e2e-ait-143fdde2-7b46869895-5x84x\" (UID: \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:36.834260 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.834220    2580 generic.go:356] "Generic (PLEG): container finished" podID="d23df273-acef-4823-9eb8-75bab1dfd75e" containerID="808115ac89d52a6c184dba52ae88776f8536301fd0951e43cd6ce1ae4691e180" exitCode=0
Jul 09 23:52:36.834430 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.834287    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"
Jul 09 23:52:36.834430 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.834308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl" event={"ID":"d23df273-acef-4823-9eb8-75bab1dfd75e","Type":"ContainerDied","Data":"808115ac89d52a6c184dba52ae88776f8536301fd0951e43cd6ce1ae4691e180"}
Jul 09 23:52:36.834430 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.834371    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl" event={"ID":"d23df273-acef-4823-9eb8-75bab1dfd75e","Type":"ContainerDied","Data":"82bba5b9db6cde60a0c898abb78e2ba2dad418529d8f160f3d7922396dabdb54"}
Jul 09 23:52:36.834430 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.834400    2580 scope.go:117] "RemoveContainer" containerID="808115ac89d52a6c184dba52ae88776f8536301fd0951e43cd6ce1ae4691e180"
Jul 09 23:52:36.843763 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.843742    2580 scope.go:117] "RemoveContainer" containerID="808115ac89d52a6c184dba52ae88776f8536301fd0951e43cd6ce1ae4691e180"
Jul 09 23:52:36.844035 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:52:36.844017    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"808115ac89d52a6c184dba52ae88776f8536301fd0951e43cd6ce1ae4691e180\": container with ID starting with 808115ac89d52a6c184dba52ae88776f8536301fd0951e43cd6ce1ae4691e180 not found: ID does not exist" containerID="808115ac89d52a6c184dba52ae88776f8536301fd0951e43cd6ce1ae4691e180"
Jul 09 23:52:36.844125 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.844043    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"808115ac89d52a6c184dba52ae88776f8536301fd0951e43cd6ce1ae4691e180"} err="failed to get container status \"808115ac89d52a6c184dba52ae88776f8536301fd0951e43cd6ce1ae4691e180\": rpc error: code = NotFound desc = could not find container \"808115ac89d52a6c184dba52ae88776f8536301fd0951e43cd6ce1ae4691e180\": container with ID starting with 808115ac89d52a6c184dba52ae88776f8536301fd0951e43cd6ce1ae4691e180 not found: ID does not exist"
Jul 09 23:52:36.855503 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.855476    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"]
Jul 09 23:52:36.875728 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:36.875703    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d2fc08b2-openshift-default-5f697fc8f7-5tkrl"]
Jul 09 23:52:37.000524 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:37.000427    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:37.141615 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:37.141586    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"]
Jul 09 23:52:37.143744 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:52:37.143712    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b25c8f_9448_44eb_9724_b4d1fcadbd96.slice/crio-ebd248aae295a088984e722059807f220d243abbbf7680dbc62cd9e298a88f42 WatchSource:0}: Error finding container ebd248aae295a088984e722059807f220d243abbbf7680dbc62cd9e298a88f42: Status 404 returned error can't find the container with id ebd248aae295a088984e722059807f220d243abbbf7680dbc62cd9e298a88f42
Jul 09 23:52:37.843254 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:37.843218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x" event={"ID":"b3b25c8f-9448-44eb-9724-b4d1fcadbd96","Type":"ContainerStarted","Data":"8e71a896b87ebd0debb70af2b908c88eade9376a17dc1b07e85441c15e556b4d"}
Jul 09 23:52:37.843428 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:37.843267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x" event={"ID":"b3b25c8f-9448-44eb-9724-b4d1fcadbd96","Type":"ContainerStarted","Data":"ebd248aae295a088984e722059807f220d243abbbf7680dbc62cd9e298a88f42"}
Jul 09 23:52:37.843428 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:37.843332    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:37.859396 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:37.859343    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x" podStartSLOduration=1.519782496 podStartE2EDuration="1.859326456s" totalImagesPullingTime="339.54396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:52:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:52:37.857874603 +0000 UTC m=+1476.001805391" watchObservedRunningTime="2026-07-09 23:52:37.859326456 +0000 UTC m=+1476.003257242"
Jul 09 23:52:38.419842 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:38.419804    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d23df273-acef-4823-9eb8-75bab1dfd75e" path="/var/lib/kubelet/pods/d23df273-acef-4823-9eb8-75bab1dfd75e/volumes"
Jul 09 23:52:42.010732 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.010695    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"]
Jul 09 23:52:42.011173 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.010944    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x" podUID="b3b25c8f-9448-44eb-9724-b4d1fcadbd96" containerName="maas-api" containerID="cri-o://8e71a896b87ebd0debb70af2b908c88eade9376a17dc1b07e85441c15e556b4d" gracePeriod=30
Jul 09 23:52:42.020619 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.020591    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:42.272075 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.272003    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:42.345370 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.345325    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-maas-api-tls\") pod \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\" (UID: \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\") "
Jul 09 23:52:42.345574 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.345418    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j8sr\" (UniqueName: \"kubernetes.io/projected/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-kube-api-access-7j8sr\") pod \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\" (UID: \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\") "
Jul 09 23:52:42.345574 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.345440    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-openshift-service-ca\") pod \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\" (UID: \"b3b25c8f-9448-44eb-9724-b4d1fcadbd96\") "
Jul 09 23:52:42.345927 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.345902    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b3b25c8f-9448-44eb-9724-b4d1fcadbd96" (UID: "b3b25c8f-9448-44eb-9724-b4d1fcadbd96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 23:52:42.347504 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.347469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-kube-api-access-7j8sr" (OuterVolumeSpecName: "kube-api-access-7j8sr") pod "b3b25c8f-9448-44eb-9724-b4d1fcadbd96" (UID: "b3b25c8f-9448-44eb-9724-b4d1fcadbd96"). InnerVolumeSpecName "kube-api-access-7j8sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:52:42.347504 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.347488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b3b25c8f-9448-44eb-9724-b4d1fcadbd96" (UID: "b3b25c8f-9448-44eb-9724-b4d1fcadbd96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 23:52:42.446807 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.446777    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-maas-api-tls\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:42.446807 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.446804    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j8sr\" (UniqueName: \"kubernetes.io/projected/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-kube-api-access-7j8sr\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:42.446807 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.446814    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3b25c8f-9448-44eb-9724-b4d1fcadbd96-openshift-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:52:42.870527 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.870491    2580 generic.go:356] "Generic (PLEG): container finished" podID="b3b25c8f-9448-44eb-9724-b4d1fcadbd96" containerID="8e71a896b87ebd0debb70af2b908c88eade9376a17dc1b07e85441c15e556b4d" exitCode=0
Jul 09 23:52:42.870719 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.870535    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"
Jul 09 23:52:42.870719 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.870551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x" event={"ID":"b3b25c8f-9448-44eb-9724-b4d1fcadbd96","Type":"ContainerDied","Data":"8e71a896b87ebd0debb70af2b908c88eade9376a17dc1b07e85441c15e556b4d"}
Jul 09 23:52:42.870719 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.870604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x" event={"ID":"b3b25c8f-9448-44eb-9724-b4d1fcadbd96","Type":"ContainerDied","Data":"ebd248aae295a088984e722059807f220d243abbbf7680dbc62cd9e298a88f42"}
Jul 09 23:52:42.870719 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.870635    2580 scope.go:117] "RemoveContainer" containerID="8e71a896b87ebd0debb70af2b908c88eade9376a17dc1b07e85441c15e556b4d"
Jul 09 23:52:42.879528 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.879502    2580 scope.go:117] "RemoveContainer" containerID="8e71a896b87ebd0debb70af2b908c88eade9376a17dc1b07e85441c15e556b4d"
Jul 09 23:52:42.879855 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:52:42.879836    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e71a896b87ebd0debb70af2b908c88eade9376a17dc1b07e85441c15e556b4d\": container with ID starting with 8e71a896b87ebd0debb70af2b908c88eade9376a17dc1b07e85441c15e556b4d not found: ID does not exist" containerID="8e71a896b87ebd0debb70af2b908c88eade9376a17dc1b07e85441c15e556b4d"
Jul 09 23:52:42.879910 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.879864    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e71a896b87ebd0debb70af2b908c88eade9376a17dc1b07e85441c15e556b4d"} err="failed to get container status \"8e71a896b87ebd0debb70af2b908c88eade9376a17dc1b07e85441c15e556b4d\": rpc error: code = NotFound desc = could not find container \"8e71a896b87ebd0debb70af2b908c88eade9376a17dc1b07e85441c15e556b4d\": container with ID starting with 8e71a896b87ebd0debb70af2b908c88eade9376a17dc1b07e85441c15e556b4d not found: ID does not exist"
Jul 09 23:52:42.886095 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.885962    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"]
Jul 09 23:52:42.887977 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:42.887957    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-143fdde2-7b46869895-5x84x"]
Jul 09 23:52:44.420008 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:44.419967    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3b25c8f-9448-44eb-9724-b4d1fcadbd96" path="/var/lib/kubelet/pods/b3b25c8f-9448-44eb-9724-b4d1fcadbd96/volumes"
Jul 09 23:52:53.620831 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.620796    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"]
Jul 09 23:52:53.621318 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.621187    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b25c8f-9448-44eb-9724-b4d1fcadbd96" containerName="maas-api"
Jul 09 23:52:53.621318 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.621203    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b25c8f-9448-44eb-9724-b4d1fcadbd96" containerName="maas-api"
Jul 09 23:52:53.621460 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.621331    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3b25c8f-9448-44eb-9724-b4d1fcadbd96" containerName="maas-api"
Jul 09 23:52:53.625810 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.625789    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.628295 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.628271    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-fd447377-openshift-default-dockercfg-qmf5q\""
Jul 09 23:52:53.628638 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.628620    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 23:52:53.637772 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.637750    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"]
Jul 09 23:52:53.741368 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.741329    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-podinfo\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.741522 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.741379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-credential-socket\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.741522 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.741436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-workload-socket\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.741522 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.741469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-envoy\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.741683 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.741521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-token\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.741683 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.741546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-workload-certs\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.741683 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.741587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6050fcd-d27a-4a83-91c9-0916382b27f7-istiod-ca-cert\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.741683 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.741608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zghbw\" (UniqueName: \"kubernetes.io/projected/b6050fcd-d27a-4a83-91c9-0916382b27f7-kube-api-access-zghbw\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.741683 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.741642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-data\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.842092 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.842053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-workload-certs\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.842092 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.842089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6050fcd-d27a-4a83-91c9-0916382b27f7-istiod-ca-cert\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.842425 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.842119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zghbw\" (UniqueName: \"kubernetes.io/projected/b6050fcd-d27a-4a83-91c9-0916382b27f7-kube-api-access-zghbw\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.842425 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.842167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-data\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.842425 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.842222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-podinfo\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.842425 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.842267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-credential-socket\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.842425 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.842295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-workload-socket\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.842425 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.842325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-envoy\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.842425 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.842363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-token\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.842793 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.842572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-data\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.842793 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.842606    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-workload-certs\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.842882 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.842822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-credential-socket\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.842927 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.842915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6050fcd-d27a-4a83-91c9-0916382b27f7-istiod-ca-cert\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.843020 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.843000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-workload-socket\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.844840 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.844809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-envoy\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.844999 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.844977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-podinfo\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.850056 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.850030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-token\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.850305 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.850283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zghbw\" (UniqueName: \"kubernetes.io/projected/b6050fcd-d27a-4a83-91c9-0916382b27f7-kube-api-access-zghbw\") pod \"e2e-derive-fd447377-openshift-default-d85d7f55-szhl7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") " pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:53.939466 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:53.939383    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:54.108819 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.108794    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"]
Jul 09 23:52:54.110769 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:52:54.110737    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6050fcd_d27a_4a83_91c9_0916382b27f7.slice/crio-9f2920fdff08c9cb1862a47978c0802a3ffd865fa81ed345d9135298d5ab2748 WatchSource:0}: Error finding container 9f2920fdff08c9cb1862a47978c0802a3ffd865fa81ed345d9135298d5ab2748: Status 404 returned error can't find the container with id 9f2920fdff08c9cb1862a47978c0802a3ffd865fa81ed345d9135298d5ab2748
Jul 09 23:52:54.113080 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.113041    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:52:54.113184 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.113134    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:52:54.113184 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.113178    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:52:54.841152 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.841120    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"]
Jul 09 23:52:54.847643 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.847617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:52:54.851006 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.850961    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-fd447377-serving-cert\""
Jul 09 23:52:54.854522 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.854495    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"]
Jul 09 23:52:54.939502 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.939462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7" event={"ID":"b6050fcd-d27a-4a83-91c9-0916382b27f7","Type":"ContainerStarted","Data":"c9b2dcb1bcd106df0bac378db3099fb68629ba05838eec8e04fa3dc18c385229"}
Jul 09 23:52:54.939716 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.939525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7" event={"ID":"b6050fcd-d27a-4a83-91c9-0916382b27f7","Type":"ContainerStarted","Data":"9f2920fdff08c9cb1862a47978c0802a3ffd865fa81ed345d9135298d5ab2748"}
Jul 09 23:52:54.950243 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.950209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb4d6c49-4255-4719-81f7-79ae868179cd-maas-api-tls\") pod \"maas-api-e2e-derive-fd447377-5999785dc9-pzrjp\" (UID: \"fb4d6c49-4255-4719-81f7-79ae868179cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:52:54.950411 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.950286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d6c49-4255-4719-81f7-79ae868179cd-openshift-service-ca\") pod \"maas-api-e2e-derive-fd447377-5999785dc9-pzrjp\" (UID: \"fb4d6c49-4255-4719-81f7-79ae868179cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:52:54.950411 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.950392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2t8v\" (UniqueName: \"kubernetes.io/projected/fb4d6c49-4255-4719-81f7-79ae868179cd-kube-api-access-f2t8v\") pod \"maas-api-e2e-derive-fd447377-5999785dc9-pzrjp\" (UID: \"fb4d6c49-4255-4719-81f7-79ae868179cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:52:54.962515 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:54.962467    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7" podStartSLOduration=1.9624516619999999 podStartE2EDuration="1.962451662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:52:54.960889944 +0000 UTC m=+1493.104820755" watchObservedRunningTime="2026-07-09 23:52:54.962451662 +0000 UTC m=+1493.106382448"
Jul 09 23:52:55.051492 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.051452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d6c49-4255-4719-81f7-79ae868179cd-openshift-service-ca\") pod \"maas-api-e2e-derive-fd447377-5999785dc9-pzrjp\" (UID: \"fb4d6c49-4255-4719-81f7-79ae868179cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:52:55.051757 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.051731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2t8v\" (UniqueName: \"kubernetes.io/projected/fb4d6c49-4255-4719-81f7-79ae868179cd-kube-api-access-f2t8v\") pod \"maas-api-e2e-derive-fd447377-5999785dc9-pzrjp\" (UID: \"fb4d6c49-4255-4719-81f7-79ae868179cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:52:55.051855 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.051788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb4d6c49-4255-4719-81f7-79ae868179cd-maas-api-tls\") pod \"maas-api-e2e-derive-fd447377-5999785dc9-pzrjp\" (UID: \"fb4d6c49-4255-4719-81f7-79ae868179cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:52:55.051918 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:52:55.051905    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-serving-cert: secret "maas-api-e2e-derive-fd447377-serving-cert" not found
Jul 09 23:52:55.051981 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:52:55.051958    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb4d6c49-4255-4719-81f7-79ae868179cd-maas-api-tls podName:fb4d6c49-4255-4719-81f7-79ae868179cd nodeName:}" failed. No retries permitted until 2026-07-09 23:52:55.551940917 +0000 UTC m=+1493.695871685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb4d6c49-4255-4719-81f7-79ae868179cd-maas-api-tls") pod "maas-api-e2e-derive-fd447377-5999785dc9-pzrjp" (UID: "fb4d6c49-4255-4719-81f7-79ae868179cd") : secret "maas-api-e2e-derive-fd447377-serving-cert" not found
Jul 09 23:52:55.052256 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.052229    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d6c49-4255-4719-81f7-79ae868179cd-openshift-service-ca\") pod \"maas-api-e2e-derive-fd447377-5999785dc9-pzrjp\" (UID: \"fb4d6c49-4255-4719-81f7-79ae868179cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:52:55.062576 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.062543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2t8v\" (UniqueName: \"kubernetes.io/projected/fb4d6c49-4255-4719-81f7-79ae868179cd-kube-api-access-f2t8v\") pod \"maas-api-e2e-derive-fd447377-5999785dc9-pzrjp\" (UID: \"fb4d6c49-4255-4719-81f7-79ae868179cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:52:55.555916 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.555882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb4d6c49-4255-4719-81f7-79ae868179cd-maas-api-tls\") pod \"maas-api-e2e-derive-fd447377-5999785dc9-pzrjp\" (UID: \"fb4d6c49-4255-4719-81f7-79ae868179cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:52:55.558293 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.558264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb4d6c49-4255-4719-81f7-79ae868179cd-maas-api-tls\") pod \"maas-api-e2e-derive-fd447377-5999785dc9-pzrjp\" (UID: \"fb4d6c49-4255-4719-81f7-79ae868179cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:52:55.764096 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.764059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:52:55.904767 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.904736    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"]
Jul 09 23:52:55.906299 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:52:55.906274    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4d6c49_4255_4719_81f7_79ae868179cd.slice/crio-7699ebf23b7a6cdeb4a22e68bbf657669bec8f9c02b2b7467bbaa564fe34890b WatchSource:0}: Error finding container 7699ebf23b7a6cdeb4a22e68bbf657669bec8f9c02b2b7467bbaa564fe34890b: Status 404 returned error can't find the container with id 7699ebf23b7a6cdeb4a22e68bbf657669bec8f9c02b2b7467bbaa564fe34890b
Jul 09 23:52:55.939580 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.939544    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:55.944903 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.944873    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:55.946491 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.946453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp" event={"ID":"fb4d6c49-4255-4719-81f7-79ae868179cd","Type":"ContainerStarted","Data":"7699ebf23b7a6cdeb4a22e68bbf657669bec8f9c02b2b7467bbaa564fe34890b"}
Jul 09 23:52:55.946783 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.946764    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:55.947432 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:55.947415    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:52:56.955726 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:56.955690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp" event={"ID":"fb4d6c49-4255-4719-81f7-79ae868179cd","Type":"ContainerStarted","Data":"c95cfa21edd4ae33a25c3dcdd32f6c317050f1f04d4dc23dde5da58ed741f7fe"}
Jul 09 23:52:56.972468 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:56.972415    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp" podStartSLOduration=2.457388974 podStartE2EDuration="2.972396565s" totalImagesPullingTime="515.007591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:52:56.970844902 +0000 UTC m=+1495.114775701" watchObservedRunningTime="2026-07-09 23:52:56.972396565 +0000 UTC m=+1495.116327350"
Jul 09 23:52:57.962720 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:57.962688    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:52:59.363054 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:59.363012    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"]
Jul 09 23:52:59.363553 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:59.363243    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7" podUID="b6050fcd-d27a-4a83-91c9-0916382b27f7" containerName="istio-proxy" containerID="cri-o://c9b2dcb1bcd106df0bac378db3099fb68629ba05838eec8e04fa3dc18c385229" gracePeriod=30
Jul 09 23:52:59.572195 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:59.572164    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"]
Jul 09 23:52:59.972730 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:52:59.972684    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp" podUID="fb4d6c49-4255-4719-81f7-79ae868179cd" containerName="maas-api" containerID="cri-o://c95cfa21edd4ae33a25c3dcdd32f6c317050f1f04d4dc23dde5da58ed741f7fe" gracePeriod=30
Jul 09 23:53:00.279142 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.279118    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:53:00.400038 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.399999    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2t8v\" (UniqueName: \"kubernetes.io/projected/fb4d6c49-4255-4719-81f7-79ae868179cd-kube-api-access-f2t8v\") pod \"fb4d6c49-4255-4719-81f7-79ae868179cd\" (UID: \"fb4d6c49-4255-4719-81f7-79ae868179cd\") "
Jul 09 23:53:00.400418 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.400063    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d6c49-4255-4719-81f7-79ae868179cd-openshift-service-ca\") pod \"fb4d6c49-4255-4719-81f7-79ae868179cd\" (UID: \"fb4d6c49-4255-4719-81f7-79ae868179cd\") "
Jul 09 23:53:00.400418 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.400105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb4d6c49-4255-4719-81f7-79ae868179cd-maas-api-tls\") pod \"fb4d6c49-4255-4719-81f7-79ae868179cd\" (UID: \"fb4d6c49-4255-4719-81f7-79ae868179cd\") "
Jul 09 23:53:00.400494 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.400446    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4d6c49-4255-4719-81f7-79ae868179cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb4d6c49-4255-4719-81f7-79ae868179cd" (UID: "fb4d6c49-4255-4719-81f7-79ae868179cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 23:53:00.402070 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.402040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4d6c49-4255-4719-81f7-79ae868179cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb4d6c49-4255-4719-81f7-79ae868179cd" (UID: "fb4d6c49-4255-4719-81f7-79ae868179cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 23:53:00.402173 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.402138    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4d6c49-4255-4719-81f7-79ae868179cd-kube-api-access-f2t8v" (OuterVolumeSpecName: "kube-api-access-f2t8v") pod "fb4d6c49-4255-4719-81f7-79ae868179cd" (UID: "fb4d6c49-4255-4719-81f7-79ae868179cd"). InnerVolumeSpecName "kube-api-access-f2t8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:53:00.501240 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.501200    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2t8v\" (UniqueName: \"kubernetes.io/projected/fb4d6c49-4255-4719-81f7-79ae868179cd-kube-api-access-f2t8v\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:53:00.501240 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.501233    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d6c49-4255-4719-81f7-79ae868179cd-openshift-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:53:00.501240 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.501245    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb4d6c49-4255-4719-81f7-79ae868179cd-maas-api-tls\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:53:00.978411 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.978372    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb4d6c49-4255-4719-81f7-79ae868179cd" containerID="c95cfa21edd4ae33a25c3dcdd32f6c317050f1f04d4dc23dde5da58ed741f7fe" exitCode=0
Jul 09 23:53:00.978618 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.978429    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"
Jul 09 23:53:00.978618 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.978454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp" event={"ID":"fb4d6c49-4255-4719-81f7-79ae868179cd","Type":"ContainerDied","Data":"c95cfa21edd4ae33a25c3dcdd32f6c317050f1f04d4dc23dde5da58ed741f7fe"}
Jul 09 23:53:00.978618 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.978504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp" event={"ID":"fb4d6c49-4255-4719-81f7-79ae868179cd","Type":"ContainerDied","Data":"7699ebf23b7a6cdeb4a22e68bbf657669bec8f9c02b2b7467bbaa564fe34890b"}
Jul 09 23:53:00.978618 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.978527    2580 scope.go:117] "RemoveContainer" containerID="c95cfa21edd4ae33a25c3dcdd32f6c317050f1f04d4dc23dde5da58ed741f7fe"
Jul 09 23:53:00.987091 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.987075    2580 scope.go:117] "RemoveContainer" containerID="c95cfa21edd4ae33a25c3dcdd32f6c317050f1f04d4dc23dde5da58ed741f7fe"
Jul 09 23:53:00.987402 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:53:00.987378    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95cfa21edd4ae33a25c3dcdd32f6c317050f1f04d4dc23dde5da58ed741f7fe\": container with ID starting with c95cfa21edd4ae33a25c3dcdd32f6c317050f1f04d4dc23dde5da58ed741f7fe not found: ID does not exist" containerID="c95cfa21edd4ae33a25c3dcdd32f6c317050f1f04d4dc23dde5da58ed741f7fe"
Jul 09 23:53:00.987470 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.987410    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95cfa21edd4ae33a25c3dcdd32f6c317050f1f04d4dc23dde5da58ed741f7fe"} err="failed to get container status \"c95cfa21edd4ae33a25c3dcdd32f6c317050f1f04d4dc23dde5da58ed741f7fe\": rpc error: code = NotFound desc = could not find container \"c95cfa21edd4ae33a25c3dcdd32f6c317050f1f04d4dc23dde5da58ed741f7fe\": container with ID starting with c95cfa21edd4ae33a25c3dcdd32f6c317050f1f04d4dc23dde5da58ed741f7fe not found: ID does not exist"
Jul 09 23:53:00.994906 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.994881    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"]
Jul 09 23:53:00.999994 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:00.999966    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fd447377-5999785dc9-pzrjp"]
Jul 09 23:53:02.420635 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:02.420599    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb4d6c49-4255-4719-81f7-79ae868179cd" path="/var/lib/kubelet/pods/fb4d6c49-4255-4719-81f7-79ae868179cd/volumes"
Jul 09 23:53:04.615259 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.615234    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:53:04.737831 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.737718    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-token\") pod \"b6050fcd-d27a-4a83-91c9-0916382b27f7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") "
Jul 09 23:53:04.737831 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.737800    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-envoy\") pod \"b6050fcd-d27a-4a83-91c9-0916382b27f7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") "
Jul 09 23:53:04.737831 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.737827    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-credential-socket\") pod \"b6050fcd-d27a-4a83-91c9-0916382b27f7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") "
Jul 09 23:53:04.738402 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.737869    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-workload-socket\") pod \"b6050fcd-d27a-4a83-91c9-0916382b27f7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") "
Jul 09 23:53:04.738402 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.737916    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-data\") pod \"b6050fcd-d27a-4a83-91c9-0916382b27f7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") "
Jul 09 23:53:04.738402 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.737940    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6050fcd-d27a-4a83-91c9-0916382b27f7-istiod-ca-cert\") pod \"b6050fcd-d27a-4a83-91c9-0916382b27f7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") "
Jul 09 23:53:04.738402 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.737982    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-workload-certs\") pod \"b6050fcd-d27a-4a83-91c9-0916382b27f7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") "
Jul 09 23:53:04.738402 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.738033    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zghbw\" (UniqueName: \"kubernetes.io/projected/b6050fcd-d27a-4a83-91c9-0916382b27f7-kube-api-access-zghbw\") pod \"b6050fcd-d27a-4a83-91c9-0916382b27f7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") "
Jul 09 23:53:04.738402 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.738080    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-podinfo\") pod \"b6050fcd-d27a-4a83-91c9-0916382b27f7\" (UID: \"b6050fcd-d27a-4a83-91c9-0916382b27f7\") "
Jul 09 23:53:04.738402 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.738114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b6050fcd-d27a-4a83-91c9-0916382b27f7" (UID: "b6050fcd-d27a-4a83-91c9-0916382b27f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:53:04.738402 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.738203    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b6050fcd-d27a-4a83-91c9-0916382b27f7" (UID: "b6050fcd-d27a-4a83-91c9-0916382b27f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:53:04.738402 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.738254    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b6050fcd-d27a-4a83-91c9-0916382b27f7" (UID: "b6050fcd-d27a-4a83-91c9-0916382b27f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:53:04.739011 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.738480    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-credential-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:53:04.739011 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.738499    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-workload-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:53:04.739011 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.738502    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6050fcd-d27a-4a83-91c9-0916382b27f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b6050fcd-d27a-4a83-91c9-0916382b27f7" (UID: "b6050fcd-d27a-4a83-91c9-0916382b27f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 23:53:04.739011 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.738513    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-workload-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:53:04.739011 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.738722    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "b6050fcd-d27a-4a83-91c9-0916382b27f7" (UID: "b6050fcd-d27a-4a83-91c9-0916382b27f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:53:04.740195 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.740164    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "b6050fcd-d27a-4a83-91c9-0916382b27f7" (UID: "b6050fcd-d27a-4a83-91c9-0916382b27f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:53:04.740293 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.740228    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6050fcd-d27a-4a83-91c9-0916382b27f7-kube-api-access-zghbw" (OuterVolumeSpecName: "kube-api-access-zghbw") pod "b6050fcd-d27a-4a83-91c9-0916382b27f7" (UID: "b6050fcd-d27a-4a83-91c9-0916382b27f7"). InnerVolumeSpecName "kube-api-access-zghbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:53:04.740293 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.740275    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b6050fcd-d27a-4a83-91c9-0916382b27f7" (UID: "b6050fcd-d27a-4a83-91c9-0916382b27f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 23:53:04.740577 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.740561    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b6050fcd-d27a-4a83-91c9-0916382b27f7" (UID: "b6050fcd-d27a-4a83-91c9-0916382b27f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:53:04.839972 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.839938    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-token\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:53:04.839972 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.839968    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-envoy\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:53:04.839972 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.839976    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-data\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:53:04.840237 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.839986    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6050fcd-d27a-4a83-91c9-0916382b27f7-istiod-ca-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:53:04.840237 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.839994    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zghbw\" (UniqueName: \"kubernetes.io/projected/b6050fcd-d27a-4a83-91c9-0916382b27f7-kube-api-access-zghbw\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:53:04.840237 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:04.840004    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6050fcd-d27a-4a83-91c9-0916382b27f7-istio-podinfo\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:53:05.001012 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:05.000925    2580 generic.go:356] "Generic (PLEG): container finished" podID="b6050fcd-d27a-4a83-91c9-0916382b27f7" containerID="c9b2dcb1bcd106df0bac378db3099fb68629ba05838eec8e04fa3dc18c385229" exitCode=0
Jul 09 23:53:05.001012 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:05.000969    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"
Jul 09 23:53:05.001212 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:05.001009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7" event={"ID":"b6050fcd-d27a-4a83-91c9-0916382b27f7","Type":"ContainerDied","Data":"c9b2dcb1bcd106df0bac378db3099fb68629ba05838eec8e04fa3dc18c385229"}
Jul 09 23:53:05.001212 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:05.001064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7" event={"ID":"b6050fcd-d27a-4a83-91c9-0916382b27f7","Type":"ContainerDied","Data":"9f2920fdff08c9cb1862a47978c0802a3ffd865fa81ed345d9135298d5ab2748"}
Jul 09 23:53:05.001212 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:05.001090    2580 scope.go:117] "RemoveContainer" containerID="c9b2dcb1bcd106df0bac378db3099fb68629ba05838eec8e04fa3dc18c385229"
Jul 09 23:53:05.017237 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:05.017207    2580 scope.go:117] "RemoveContainer" containerID="c9b2dcb1bcd106df0bac378db3099fb68629ba05838eec8e04fa3dc18c385229"
Jul 09 23:53:05.017539 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:53:05.017521    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b2dcb1bcd106df0bac378db3099fb68629ba05838eec8e04fa3dc18c385229\": container with ID starting with c9b2dcb1bcd106df0bac378db3099fb68629ba05838eec8e04fa3dc18c385229 not found: ID does not exist" containerID="c9b2dcb1bcd106df0bac378db3099fb68629ba05838eec8e04fa3dc18c385229"
Jul 09 23:53:05.017596 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:05.017547    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b2dcb1bcd106df0bac378db3099fb68629ba05838eec8e04fa3dc18c385229"} err="failed to get container status \"c9b2dcb1bcd106df0bac378db3099fb68629ba05838eec8e04fa3dc18c385229\": rpc error: code = NotFound desc = could not find container \"c9b2dcb1bcd106df0bac378db3099fb68629ba05838eec8e04fa3dc18c385229\": container with ID starting with c9b2dcb1bcd106df0bac378db3099fb68629ba05838eec8e04fa3dc18c385229 not found: ID does not exist"
Jul 09 23:53:05.027072 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:05.027043    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"]
Jul 09 23:53:05.029950 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:05.029923    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-fd447377-openshift-default-d85d7f55-szhl7"]
Jul 09 23:53:06.419854 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:53:06.419817    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6050fcd-d27a-4a83-91c9-0916382b27f7" path="/var/lib/kubelet/pods/b6050fcd-d27a-4a83-91c9-0916382b27f7/volumes"
Jul 09 23:54:39.061067 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.061031    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"]
Jul 09 23:54:39.061527 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.061386    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6050fcd-d27a-4a83-91c9-0916382b27f7" containerName="istio-proxy"
Jul 09 23:54:39.061527 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.061397    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6050fcd-d27a-4a83-91c9-0916382b27f7" containerName="istio-proxy"
Jul 09 23:54:39.061527 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.061415    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4d6c49-4255-4719-81f7-79ae868179cd" containerName="maas-api"
Jul 09 23:54:39.061527 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.061421    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4d6c49-4255-4719-81f7-79ae868179cd" containerName="maas-api"
Jul 09 23:54:39.061527 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.061481    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6050fcd-d27a-4a83-91c9-0916382b27f7" containerName="istio-proxy"
Jul 09 23:54:39.061527 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.061489    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4d6c49-4255-4719-81f7-79ae868179cd" containerName="maas-api"
Jul 09 23:54:39.064386 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.064370    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.066790 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.066766    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 23:54:39.066916 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.066766    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-da6e813e-openshift-default-dockercfg-tnc24\""
Jul 09 23:54:39.084869 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.084838    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"]
Jul 09 23:54:39.214832 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.214790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-podinfo\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.214832 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.214840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-envoy\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.215088 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.214940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-data\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.215088 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.214982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-workload-certs\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.215088 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.215017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-token\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.215088 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.215063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nznr6\" (UniqueName: \"kubernetes.io/projected/d845f196-346f-4e90-b3c7-446b21f52ec5-kube-api-access-nznr6\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.215253 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.215092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-workload-socket\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.215253 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.215110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d845f196-346f-4e90-b3c7-446b21f52ec5-istiod-ca-cert\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.215253 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.215134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-credential-socket\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.316461 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.316371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-workload-certs\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.316461 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.316415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-token\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.316741 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.316504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nznr6\" (UniqueName: \"kubernetes.io/projected/d845f196-346f-4e90-b3c7-446b21f52ec5-kube-api-access-nznr6\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.316741 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.316522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-workload-socket\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.316741 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.316539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d845f196-346f-4e90-b3c7-446b21f52ec5-istiod-ca-cert\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.316741 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.316562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-credential-socket\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.316741 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.316595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-podinfo\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.316741 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.316619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-envoy\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.317045 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.316779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-workload-certs\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.317045 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.316846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-data\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.317045 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.316898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-workload-socket\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.317156 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.317044    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-credential-socket\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.317156 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.317145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-data\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.317270 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.317250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d845f196-346f-4e90-b3c7-446b21f52ec5-istiod-ca-cert\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.318854 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.318836    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-envoy\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.319065 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.319049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-podinfo\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.324579 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.324555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-token\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.324897 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.324876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nznr6\" (UniqueName: \"kubernetes.io/projected/d845f196-346f-4e90-b3c7-446b21f52ec5-kube-api-access-nznr6\") pod \"e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") " pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.374791 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.374755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:39.519845 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.519816    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"]
Jul 09 23:54:39.521779 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:54:39.521751    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd845f196_346f_4e90_b3c7_446b21f52ec5.slice/crio-95e29678b85af0648dfc771a401c91aaa272d8e721254e072538747699575267 WatchSource:0}: Error finding container 95e29678b85af0648dfc771a401c91aaa272d8e721254e072538747699575267: Status 404 returned error can't find the container with id 95e29678b85af0648dfc771a401c91aaa272d8e721254e072538747699575267
Jul 09 23:54:39.523951 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.523906    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:54:39.524031 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.524012    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:54:39.524076 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:39.524059    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:54:40.500719 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:40.500679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj" event={"ID":"d845f196-346f-4e90-b3c7-446b21f52ec5","Type":"ContainerStarted","Data":"7cebac98185526a89095f57150a4e6c8cd08ab411eadd7b994869b118d9b9383"}
Jul 09 23:54:40.501162 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:40.500732    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj" event={"ID":"d845f196-346f-4e90-b3c7-446b21f52ec5","Type":"ContainerStarted","Data":"95e29678b85af0648dfc771a401c91aaa272d8e721254e072538747699575267"}
Jul 09 23:54:40.520276 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:40.520224    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj" podStartSLOduration=1.520206764 podStartE2EDuration="1.520206764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:54:40.518421361 +0000 UTC m=+1598.662352147" watchObservedRunningTime="2026-07-09 23:54:40.520206764 +0000 UTC m=+1598.664137549"
Jul 09 23:54:41.375179 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:41.375143    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:41.380025 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:41.379996    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:41.505677 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:41.505620    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:41.506786 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:41.506771    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:42.489980 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:42.489945    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"]
Jul 09 23:54:42.493382 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:42.493357    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:42.495963 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:42.495941    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-da6e813e-serving-cert\""
Jul 09 23:54:42.504634 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:42.504608    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"]
Jul 09 23:54:42.651270 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:42.651234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536f243e-bdcb-4371-adde-e7c7ef49ee79-maas-api-tls\") pod \"maas-api-e2e-mt-da6e813e-76df8b8844-lldgv\" (UID: \"536f243e-bdcb-4371-adde-e7c7ef49ee79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:42.651729 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:42.651294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536f243e-bdcb-4371-adde-e7c7ef49ee79-openshift-service-ca\") pod \"maas-api-e2e-mt-da6e813e-76df8b8844-lldgv\" (UID: \"536f243e-bdcb-4371-adde-e7c7ef49ee79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:42.651729 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:42.651402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb448\" (UniqueName: \"kubernetes.io/projected/536f243e-bdcb-4371-adde-e7c7ef49ee79-kube-api-access-fb448\") pod \"maas-api-e2e-mt-da6e813e-76df8b8844-lldgv\" (UID: \"536f243e-bdcb-4371-adde-e7c7ef49ee79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:42.752894 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:42.752799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536f243e-bdcb-4371-adde-e7c7ef49ee79-maas-api-tls\") pod \"maas-api-e2e-mt-da6e813e-76df8b8844-lldgv\" (UID: \"536f243e-bdcb-4371-adde-e7c7ef49ee79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:42.752894 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:54:42.752840    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-serving-cert: secret "maas-api-e2e-mt-da6e813e-serving-cert" not found
Jul 09 23:54:42.753101 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:54:42.752909    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/536f243e-bdcb-4371-adde-e7c7ef49ee79-maas-api-tls podName:536f243e-bdcb-4371-adde-e7c7ef49ee79 nodeName:}" failed. No retries permitted until 2026-07-09 23:54:43.252891457 +0000 UTC m=+1601.396822226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/536f243e-bdcb-4371-adde-e7c7ef49ee79-maas-api-tls") pod "maas-api-e2e-mt-da6e813e-76df8b8844-lldgv" (UID: "536f243e-bdcb-4371-adde-e7c7ef49ee79") : secret "maas-api-e2e-mt-da6e813e-serving-cert" not found
Jul 09 23:54:42.753101 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:42.752850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536f243e-bdcb-4371-adde-e7c7ef49ee79-openshift-service-ca\") pod \"maas-api-e2e-mt-da6e813e-76df8b8844-lldgv\" (UID: \"536f243e-bdcb-4371-adde-e7c7ef49ee79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:42.753101 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:42.753020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb448\" (UniqueName: \"kubernetes.io/projected/536f243e-bdcb-4371-adde-e7c7ef49ee79-kube-api-access-fb448\") pod \"maas-api-e2e-mt-da6e813e-76df8b8844-lldgv\" (UID: \"536f243e-bdcb-4371-adde-e7c7ef49ee79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:42.753405 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:42.753389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536f243e-bdcb-4371-adde-e7c7ef49ee79-openshift-service-ca\") pod \"maas-api-e2e-mt-da6e813e-76df8b8844-lldgv\" (UID: \"536f243e-bdcb-4371-adde-e7c7ef49ee79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:42.762139 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:42.762109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb448\" (UniqueName: \"kubernetes.io/projected/536f243e-bdcb-4371-adde-e7c7ef49ee79-kube-api-access-fb448\") pod \"maas-api-e2e-mt-da6e813e-76df8b8844-lldgv\" (UID: \"536f243e-bdcb-4371-adde-e7c7ef49ee79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:43.256662 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:43.256612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536f243e-bdcb-4371-adde-e7c7ef49ee79-maas-api-tls\") pod \"maas-api-e2e-mt-da6e813e-76df8b8844-lldgv\" (UID: \"536f243e-bdcb-4371-adde-e7c7ef49ee79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:43.259049 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:43.259029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536f243e-bdcb-4371-adde-e7c7ef49ee79-maas-api-tls\") pod \"maas-api-e2e-mt-da6e813e-76df8b8844-lldgv\" (UID: \"536f243e-bdcb-4371-adde-e7c7ef49ee79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:43.405704 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:43.405636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:43.546008 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:43.545984    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"]
Jul 09 23:54:43.547909 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:54:43.547874    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536f243e_bdcb_4371_adde_e7c7ef49ee79.slice/crio-acfa9394d4ed83480653012fcfc64da226bec4c74448d02c4826898f9556def3 WatchSource:0}: Error finding container acfa9394d4ed83480653012fcfc64da226bec4c74448d02c4826898f9556def3: Status 404 returned error can't find the container with id acfa9394d4ed83480653012fcfc64da226bec4c74448d02c4826898f9556def3
Jul 09 23:54:44.527342 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:44.527307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv" event={"ID":"536f243e-bdcb-4371-adde-e7c7ef49ee79","Type":"ContainerStarted","Data":"975ad899dc1e01ea16205530df84b5fd148721ede4530c16be2064fb16afe027"}
Jul 09 23:54:44.527850 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:44.527366    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv" event={"ID":"536f243e-bdcb-4371-adde-e7c7ef49ee79","Type":"ContainerStarted","Data":"acfa9394d4ed83480653012fcfc64da226bec4c74448d02c4826898f9556def3"}
Jul 09 23:54:44.527850 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:44.527411    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:44.544484 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:44.544432    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv" podStartSLOduration=2.15592883 podStartE2EDuration="2.544416678s" totalImagesPullingTime="388.487848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:54:44.542675511 +0000 UTC m=+1602.686606299" watchObservedRunningTime="2026-07-09 23:54:44.544416678 +0000 UTC m=+1602.688347466"
Jul 09 23:54:50.548019 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:50.547976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:54.563169 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:54.563135    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"]
Jul 09 23:54:54.563569 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:54.563381    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv" podUID="536f243e-bdcb-4371-adde-e7c7ef49ee79" containerName="maas-api" containerID="cri-o://975ad899dc1e01ea16205530df84b5fd148721ede4530c16be2064fb16afe027" gracePeriod=30
Jul 09 23:54:54.820118 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:54.820088    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:54.865284 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:54.865255    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536f243e-bdcb-4371-adde-e7c7ef49ee79-openshift-service-ca\") pod \"536f243e-bdcb-4371-adde-e7c7ef49ee79\" (UID: \"536f243e-bdcb-4371-adde-e7c7ef49ee79\") "
Jul 09 23:54:54.865477 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:54.865308    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb448\" (UniqueName: \"kubernetes.io/projected/536f243e-bdcb-4371-adde-e7c7ef49ee79-kube-api-access-fb448\") pod \"536f243e-bdcb-4371-adde-e7c7ef49ee79\" (UID: \"536f243e-bdcb-4371-adde-e7c7ef49ee79\") "
Jul 09 23:54:54.865477 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:54.865450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536f243e-bdcb-4371-adde-e7c7ef49ee79-maas-api-tls\") pod \"536f243e-bdcb-4371-adde-e7c7ef49ee79\" (UID: \"536f243e-bdcb-4371-adde-e7c7ef49ee79\") "
Jul 09 23:54:54.865696 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:54.865670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536f243e-bdcb-4371-adde-e7c7ef49ee79-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "536f243e-bdcb-4371-adde-e7c7ef49ee79" (UID: "536f243e-bdcb-4371-adde-e7c7ef49ee79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 23:54:54.867386 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:54.867357    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536f243e-bdcb-4371-adde-e7c7ef49ee79-kube-api-access-fb448" (OuterVolumeSpecName: "kube-api-access-fb448") pod "536f243e-bdcb-4371-adde-e7c7ef49ee79" (UID: "536f243e-bdcb-4371-adde-e7c7ef49ee79"). InnerVolumeSpecName "kube-api-access-fb448". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:54:54.867592 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:54.867561    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536f243e-bdcb-4371-adde-e7c7ef49ee79-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "536f243e-bdcb-4371-adde-e7c7ef49ee79" (UID: "536f243e-bdcb-4371-adde-e7c7ef49ee79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 23:54:54.966854 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:54.966821    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536f243e-bdcb-4371-adde-e7c7ef49ee79-openshift-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:54:54.966854 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:54.966852    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb448\" (UniqueName: \"kubernetes.io/projected/536f243e-bdcb-4371-adde-e7c7ef49ee79-kube-api-access-fb448\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:54:54.966854 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:54.966862    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536f243e-bdcb-4371-adde-e7c7ef49ee79-maas-api-tls\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:54:55.591003 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:55.590965    2580 generic.go:356] "Generic (PLEG): container finished" podID="536f243e-bdcb-4371-adde-e7c7ef49ee79" containerID="975ad899dc1e01ea16205530df84b5fd148721ede4530c16be2064fb16afe027" exitCode=0
Jul 09 23:54:55.591415 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:55.591011    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"
Jul 09 23:54:55.591415 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:55.591050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv" event={"ID":"536f243e-bdcb-4371-adde-e7c7ef49ee79","Type":"ContainerDied","Data":"975ad899dc1e01ea16205530df84b5fd148721ede4530c16be2064fb16afe027"}
Jul 09 23:54:55.591415 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:55.591109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv" event={"ID":"536f243e-bdcb-4371-adde-e7c7ef49ee79","Type":"ContainerDied","Data":"acfa9394d4ed83480653012fcfc64da226bec4c74448d02c4826898f9556def3"}
Jul 09 23:54:55.591415 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:55.591137    2580 scope.go:117] "RemoveContainer" containerID="975ad899dc1e01ea16205530df84b5fd148721ede4530c16be2064fb16afe027"
Jul 09 23:54:55.600935 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:55.600892    2580 scope.go:117] "RemoveContainer" containerID="975ad899dc1e01ea16205530df84b5fd148721ede4530c16be2064fb16afe027"
Jul 09 23:54:55.601220 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:54:55.601194    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975ad899dc1e01ea16205530df84b5fd148721ede4530c16be2064fb16afe027\": container with ID starting with 975ad899dc1e01ea16205530df84b5fd148721ede4530c16be2064fb16afe027 not found: ID does not exist" containerID="975ad899dc1e01ea16205530df84b5fd148721ede4530c16be2064fb16afe027"
Jul 09 23:54:55.601318 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:55.601228    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975ad899dc1e01ea16205530df84b5fd148721ede4530c16be2064fb16afe027"} err="failed to get container status \"975ad899dc1e01ea16205530df84b5fd148721ede4530c16be2064fb16afe027\": rpc error: code = NotFound desc = could not find container \"975ad899dc1e01ea16205530df84b5fd148721ede4530c16be2064fb16afe027\": container with ID starting with 975ad899dc1e01ea16205530df84b5fd148721ede4530c16be2064fb16afe027 not found: ID does not exist"
Jul 09 23:54:55.612529 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:55.612504    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"]
Jul 09 23:54:55.615229 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:55.615196    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-da6e813e-76df8b8844-lldgv"]
Jul 09 23:54:56.419569 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:56.419534    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536f243e-bdcb-4371-adde-e7c7ef49ee79" path="/var/lib/kubelet/pods/536f243e-bdcb-4371-adde-e7c7ef49ee79/volumes"
Jul 09 23:54:57.319098 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:57.318596    2580 patch_prober.go:28] interesting pod/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:58606->10.134.0.57:15021: read: connection reset by peer" start-of-body=
Jul 09 23:54:57.319098 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:57.318682    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj" podUID="d845f196-346f-4e90-b3c7-446b21f52ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:58606->10.134.0.57:15021: read: connection reset by peer"
Jul 09 23:54:57.604874 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:57.604788    2580 generic.go:356] "Generic (PLEG): container finished" podID="d845f196-346f-4e90-b3c7-446b21f52ec5" containerID="7cebac98185526a89095f57150a4e6c8cd08ab411eadd7b994869b118d9b9383" exitCode=0
Jul 09 23:54:57.604874 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:57.604862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj" event={"ID":"d845f196-346f-4e90-b3c7-446b21f52ec5","Type":"ContainerDied","Data":"7cebac98185526a89095f57150a4e6c8cd08ab411eadd7b994869b118d9b9383"}
Jul 09 23:54:57.605311 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:57.605291    2580 scope.go:117] "RemoveContainer" containerID="7cebac98185526a89095f57150a4e6c8cd08ab411eadd7b994869b118d9b9383"
Jul 09 23:54:57.606295 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:57.606265    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:54:57.606388 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:57.606352    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:54:57.606450 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:57.606401    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:54:58.612268 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:58.612235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj" event={"ID":"d845f196-346f-4e90-b3c7-446b21f52ec5","Type":"ContainerStarted","Data":"c8318d2397fd0501e5b780fe265b923f1bf72fc3ab4cd4616a87f023daa35579"}
Jul 09 23:54:59.375176 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:59.375138    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:54:59.375414 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:59.375389    2580 patch_prober.go:28] interesting pod/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 09 23:54:59.375471 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:54:59.375451    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj" podUID="d845f196-346f-4e90-b3c7-446b21f52ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 09 23:55:00.375159 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:00.375114    2580 patch_prober.go:28] interesting pod/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 09 23:55:00.375642 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:00.375187    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj" podUID="d845f196-346f-4e90-b3c7-446b21f52ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 09 23:55:00.431679 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:00.431555    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"]
Jul 09 23:55:00.623197 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:00.623158    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj" podUID="d845f196-346f-4e90-b3c7-446b21f52ec5" containerName="istio-proxy" containerID="cri-o://c8318d2397fd0501e5b780fe265b923f1bf72fc3ab4cd4616a87f023daa35579" gracePeriod=30
Jul 09 23:55:08.185259 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.185232    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:55:08.288846 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.288799    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-workload-socket\") pod \"d845f196-346f-4e90-b3c7-446b21f52ec5\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") "
Jul 09 23:55:08.289028 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.288871    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-data\") pod \"d845f196-346f-4e90-b3c7-446b21f52ec5\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") "
Jul 09 23:55:08.289028 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.288894    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d845f196-346f-4e90-b3c7-446b21f52ec5-istiod-ca-cert\") pod \"d845f196-346f-4e90-b3c7-446b21f52ec5\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") "
Jul 09 23:55:08.289028 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.288918    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-workload-certs\") pod \"d845f196-346f-4e90-b3c7-446b21f52ec5\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") "
Jul 09 23:55:08.289028 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.288939    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nznr6\" (UniqueName: \"kubernetes.io/projected/d845f196-346f-4e90-b3c7-446b21f52ec5-kube-api-access-nznr6\") pod \"d845f196-346f-4e90-b3c7-446b21f52ec5\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") "
Jul 09 23:55:08.289226 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.289029    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-credential-socket\") pod \"d845f196-346f-4e90-b3c7-446b21f52ec5\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") "
Jul 09 23:55:08.289226 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.289092    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-token\") pod \"d845f196-346f-4e90-b3c7-446b21f52ec5\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") "
Jul 09 23:55:08.289226 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.289122    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-podinfo\") pod \"d845f196-346f-4e90-b3c7-446b21f52ec5\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") "
Jul 09 23:55:08.289226 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.289137    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d845f196-346f-4e90-b3c7-446b21f52ec5" (UID: "d845f196-346f-4e90-b3c7-446b21f52ec5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:55:08.289226 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.289163    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-envoy\") pod \"d845f196-346f-4e90-b3c7-446b21f52ec5\" (UID: \"d845f196-346f-4e90-b3c7-446b21f52ec5\") "
Jul 09 23:55:08.289486 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.289249    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d845f196-346f-4e90-b3c7-446b21f52ec5" (UID: "d845f196-346f-4e90-b3c7-446b21f52ec5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:55:08.289486 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.289385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d845f196-346f-4e90-b3c7-446b21f52ec5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d845f196-346f-4e90-b3c7-446b21f52ec5" (UID: "d845f196-346f-4e90-b3c7-446b21f52ec5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 23:55:08.289486 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.289450    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-workload-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:55:08.289486 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.289466    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d845f196-346f-4e90-b3c7-446b21f52ec5-istiod-ca-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:55:08.289486 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.289480    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-workload-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:55:08.289776 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.289643    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d845f196-346f-4e90-b3c7-446b21f52ec5" (UID: "d845f196-346f-4e90-b3c7-446b21f52ec5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:55:08.289898 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.289873    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-data" (OuterVolumeSpecName: "istio-data") pod "d845f196-346f-4e90-b3c7-446b21f52ec5" (UID: "d845f196-346f-4e90-b3c7-446b21f52ec5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:55:08.291224 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.291187    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d845f196-346f-4e90-b3c7-446b21f52ec5" (UID: "d845f196-346f-4e90-b3c7-446b21f52ec5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:55:08.291351 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.291301    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d845f196-346f-4e90-b3c7-446b21f52ec5-kube-api-access-nznr6" (OuterVolumeSpecName: "kube-api-access-nznr6") pod "d845f196-346f-4e90-b3c7-446b21f52ec5" (UID: "d845f196-346f-4e90-b3c7-446b21f52ec5"). InnerVolumeSpecName "kube-api-access-nznr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:55:08.291540 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.291521    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-token" (OuterVolumeSpecName: "istio-token") pod "d845f196-346f-4e90-b3c7-446b21f52ec5" (UID: "d845f196-346f-4e90-b3c7-446b21f52ec5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:55:08.291588 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.291546    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d845f196-346f-4e90-b3c7-446b21f52ec5" (UID: "d845f196-346f-4e90-b3c7-446b21f52ec5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 23:55:08.390423 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.390378    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-credential-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:55:08.390423 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.390414    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-token\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:55:08.390423 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.390424    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-podinfo\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:55:08.390423 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.390432    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-envoy\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:55:08.390732 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.390440    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d845f196-346f-4e90-b3c7-446b21f52ec5-istio-data\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:55:08.390732 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.390450    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nznr6\" (UniqueName: \"kubernetes.io/projected/d845f196-346f-4e90-b3c7-446b21f52ec5-kube-api-access-nznr6\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:55:08.673703 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.673597    2580 generic.go:356] "Generic (PLEG): container finished" podID="d845f196-346f-4e90-b3c7-446b21f52ec5" containerID="c8318d2397fd0501e5b780fe265b923f1bf72fc3ab4cd4616a87f023daa35579" exitCode=0
Jul 09 23:55:08.673703 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.673661    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"
Jul 09 23:55:08.673703 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.673683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj" event={"ID":"d845f196-346f-4e90-b3c7-446b21f52ec5","Type":"ContainerDied","Data":"c8318d2397fd0501e5b780fe265b923f1bf72fc3ab4cd4616a87f023daa35579"}
Jul 09 23:55:08.673991 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.673776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj" event={"ID":"d845f196-346f-4e90-b3c7-446b21f52ec5","Type":"ContainerDied","Data":"95e29678b85af0648dfc771a401c91aaa272d8e721254e072538747699575267"}
Jul 09 23:55:08.673991 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.673807    2580 scope.go:117] "RemoveContainer" containerID="c8318d2397fd0501e5b780fe265b923f1bf72fc3ab4cd4616a87f023daa35579"
Jul 09 23:55:08.683541 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.683515    2580 scope.go:117] "RemoveContainer" containerID="7cebac98185526a89095f57150a4e6c8cd08ab411eadd7b994869b118d9b9383"
Jul 09 23:55:08.691620 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.691592    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"]
Jul 09 23:55:08.692016 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.692003    2580 scope.go:117] "RemoveContainer" containerID="c8318d2397fd0501e5b780fe265b923f1bf72fc3ab4cd4616a87f023daa35579"
Jul 09 23:55:08.692294 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:55:08.692274    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8318d2397fd0501e5b780fe265b923f1bf72fc3ab4cd4616a87f023daa35579\": container with ID starting with c8318d2397fd0501e5b780fe265b923f1bf72fc3ab4cd4616a87f023daa35579 not found: ID does not exist" containerID="c8318d2397fd0501e5b780fe265b923f1bf72fc3ab4cd4616a87f023daa35579"
Jul 09 23:55:08.692363 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.692301    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8318d2397fd0501e5b780fe265b923f1bf72fc3ab4cd4616a87f023daa35579"} err="failed to get container status \"c8318d2397fd0501e5b780fe265b923f1bf72fc3ab4cd4616a87f023daa35579\": rpc error: code = NotFound desc = could not find container \"c8318d2397fd0501e5b780fe265b923f1bf72fc3ab4cd4616a87f023daa35579\": container with ID starting with c8318d2397fd0501e5b780fe265b923f1bf72fc3ab4cd4616a87f023daa35579 not found: ID does not exist"
Jul 09 23:55:08.692363 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.692319    2580 scope.go:117] "RemoveContainer" containerID="7cebac98185526a89095f57150a4e6c8cd08ab411eadd7b994869b118d9b9383"
Jul 09 23:55:08.695775 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:55:08.695737    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cebac98185526a89095f57150a4e6c8cd08ab411eadd7b994869b118d9b9383\": container with ID starting with 7cebac98185526a89095f57150a4e6c8cd08ab411eadd7b994869b118d9b9383 not found: ID does not exist" containerID="7cebac98185526a89095f57150a4e6c8cd08ab411eadd7b994869b118d9b9383"
Jul 09 23:55:08.695878 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.695796    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cebac98185526a89095f57150a4e6c8cd08ab411eadd7b994869b118d9b9383"} err="failed to get container status \"7cebac98185526a89095f57150a4e6c8cd08ab411eadd7b994869b118d9b9383\": rpc error: code = NotFound desc = could not find container \"7cebac98185526a89095f57150a4e6c8cd08ab411eadd7b994869b118d9b9383\": container with ID starting with 7cebac98185526a89095f57150a4e6c8cd08ab411eadd7b994869b118d9b9383 not found: ID does not exist"
Jul 09 23:55:08.698100 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:08.698077    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-da6e813e-openshift-default-75cd7df4cf-98qwj"]
Jul 09 23:55:10.419269 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:10.419233    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d845f196-346f-4e90-b3c7-446b21f52ec5" path="/var/lib/kubelet/pods/d845f196-346f-4e90-b3c7-446b21f52ec5/volumes"
Jul 09 23:55:19.606531 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.606499    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"]
Jul 09 23:55:19.607006 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.606958    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d845f196-346f-4e90-b3c7-446b21f52ec5" containerName="istio-proxy"
Jul 09 23:55:19.607006 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.606977    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d845f196-346f-4e90-b3c7-446b21f52ec5" containerName="istio-proxy"
Jul 09 23:55:19.607006 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.607001    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d845f196-346f-4e90-b3c7-446b21f52ec5" containerName="istio-proxy"
Jul 09 23:55:19.607178 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.607010    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d845f196-346f-4e90-b3c7-446b21f52ec5" containerName="istio-proxy"
Jul 09 23:55:19.607178 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.607022    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536f243e-bdcb-4371-adde-e7c7ef49ee79" containerName="maas-api"
Jul 09 23:55:19.607178 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.607030    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="536f243e-bdcb-4371-adde-e7c7ef49ee79" containerName="maas-api"
Jul 09 23:55:19.607178 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.607143    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d845f196-346f-4e90-b3c7-446b21f52ec5" containerName="istio-proxy"
Jul 09 23:55:19.607178 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.607155    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="536f243e-bdcb-4371-adde-e7c7ef49ee79" containerName="maas-api"
Jul 09 23:55:19.611768 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.611743    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.614398 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.614372    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7b0dea16-openshift-default-dockercfg-s6hjc\""
Jul 09 23:55:19.614398 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.614372    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 23:55:19.622843 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.622819    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"]
Jul 09 23:55:19.687296 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.687256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63610221-9416-4944-b620-fab17c28b90b-istiod-ca-cert\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.687487 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.687324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-workload-socket\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.687487 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.687353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-credential-socket\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.687487 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.687379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f825k\" (UniqueName: \"kubernetes.io/projected/63610221-9416-4944-b620-fab17c28b90b-kube-api-access-f825k\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.687487 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.687408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-istio-data\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.687487 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.687451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63610221-9416-4944-b620-fab17c28b90b-istio-token\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.687686 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.687531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63610221-9416-4944-b620-fab17c28b90b-istio-podinfo\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.687686 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.687566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-workload-certs\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.687686 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.687586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-istio-envoy\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.788375 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.788343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-workload-socket\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.788375 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.788373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-credential-socket\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.788593 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.788406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f825k\" (UniqueName: \"kubernetes.io/projected/63610221-9416-4944-b620-fab17c28b90b-kube-api-access-f825k\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.788593 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.788433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-istio-data\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.788593 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.788467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63610221-9416-4944-b620-fab17c28b90b-istio-token\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.788593 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.788515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63610221-9416-4944-b620-fab17c28b90b-istio-podinfo\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.788593 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.788542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-workload-certs\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.788593 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.788562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-istio-envoy\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.788951 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.788616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63610221-9416-4944-b620-fab17c28b90b-istiod-ca-cert\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.788951 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.788756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-workload-socket\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.788951 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.788845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-credential-socket\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.788951 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.788868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-istio-data\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.789120 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.789001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-workload-certs\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.789367 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.789348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63610221-9416-4944-b620-fab17c28b90b-istiod-ca-cert\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.790886 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.790863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-istio-envoy\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.791086 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.791067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63610221-9416-4944-b620-fab17c28b90b-istio-podinfo\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.798968 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.798945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63610221-9416-4944-b620-fab17c28b90b-istio-token\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.799173 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.799154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f825k\" (UniqueName: \"kubernetes.io/projected/63610221-9416-4944-b620-fab17c28b90b-kube-api-access-f825k\") pod \"e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") " pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:19.922889 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:19.922799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:20.059716 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:20.059691    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"]
Jul 09 23:55:20.060841 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:55:20.060811    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63610221_9416_4944_b620_fab17c28b90b.slice/crio-f7d8402c1d46fe2204033439798b8a32cd9536338ea8aa1bdf6fa99e523f26ad WatchSource:0}: Error finding container f7d8402c1d46fe2204033439798b8a32cd9536338ea8aa1bdf6fa99e523f26ad: Status 404 returned error can't find the container with id f7d8402c1d46fe2204033439798b8a32cd9536338ea8aa1bdf6fa99e523f26ad
Jul 09 23:55:20.063035 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:20.063007    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:55:20.063137 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:20.063081    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:55:20.063137 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:20.063126    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:55:20.733803 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:20.733767    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" event={"ID":"63610221-9416-4944-b620-fab17c28b90b","Type":"ContainerStarted","Data":"ed7f4b0a2670d7a1e492e9dceedd04ccbe96b833e868f35de5a51b38a3a0ed03"}
Jul 09 23:55:20.734270 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:20.733823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" event={"ID":"63610221-9416-4944-b620-fab17c28b90b","Type":"ContainerStarted","Data":"f7d8402c1d46fe2204033439798b8a32cd9536338ea8aa1bdf6fa99e523f26ad"}
Jul 09 23:55:20.752786 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:20.752735    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" podStartSLOduration=1.752719012 podStartE2EDuration="1.752719012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:55:20.751883691 +0000 UTC m=+1638.895814476" watchObservedRunningTime="2026-07-09 23:55:20.752719012 +0000 UTC m=+1638.896649797"
Jul 09 23:55:20.923665 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:20.923621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:21.927975 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:21.927941    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:22.723581 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.723541    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"]
Jul 09 23:55:22.724250 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.724229    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d845f196-346f-4e90-b3c7-446b21f52ec5" containerName="istio-proxy"
Jul 09 23:55:22.727724 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.727704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:22.730266 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.730242    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7b0dea16-serving-cert\""
Jul 09 23:55:22.739405 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.739190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"]
Jul 09 23:55:22.745340 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.745307    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:22.746439 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.746418    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:22.816575 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.816540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb75bb75-ac24-4df3-ab47-bf05e6256f73-openshift-service-ca\") pod \"maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2\" (UID: \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:22.816807 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.816671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb75bb75-ac24-4df3-ab47-bf05e6256f73-maas-api-tls\") pod \"maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2\" (UID: \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:22.816807 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.816712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p584n\" (UniqueName: \"kubernetes.io/projected/fb75bb75-ac24-4df3-ab47-bf05e6256f73-kube-api-access-p584n\") pod \"maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2\" (UID: \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:22.917853 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.917812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb75bb75-ac24-4df3-ab47-bf05e6256f73-openshift-service-ca\") pod \"maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2\" (UID: \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:22.918052 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.917859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb75bb75-ac24-4df3-ab47-bf05e6256f73-maas-api-tls\") pod \"maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2\" (UID: \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:22.918052 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.917890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p584n\" (UniqueName: \"kubernetes.io/projected/fb75bb75-ac24-4df3-ab47-bf05e6256f73-kube-api-access-p584n\") pod \"maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2\" (UID: \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:22.918052 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:55:22.918016    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-serving-cert: secret "maas-api-e2e-mt-7b0dea16-serving-cert" not found
Jul 09 23:55:22.918224 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:55:22.918092    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb75bb75-ac24-4df3-ab47-bf05e6256f73-maas-api-tls podName:fb75bb75-ac24-4df3-ab47-bf05e6256f73 nodeName:}" failed. No retries permitted until 2026-07-09 23:55:23.418075604 +0000 UTC m=+1641.562006368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb75bb75-ac24-4df3-ab47-bf05e6256f73-maas-api-tls") pod "maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2" (UID: "fb75bb75-ac24-4df3-ab47-bf05e6256f73") : secret "maas-api-e2e-mt-7b0dea16-serving-cert" not found
Jul 09 23:55:22.918471 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.918449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb75bb75-ac24-4df3-ab47-bf05e6256f73-openshift-service-ca\") pod \"maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2\" (UID: \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:22.926402 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:22.926376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p584n\" (UniqueName: \"kubernetes.io/projected/fb75bb75-ac24-4df3-ab47-bf05e6256f73-kube-api-access-p584n\") pod \"maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2\" (UID: \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:23.422033 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:23.421994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb75bb75-ac24-4df3-ab47-bf05e6256f73-maas-api-tls\") pod \"maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2\" (UID: \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:23.424394 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:23.424374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb75bb75-ac24-4df3-ab47-bf05e6256f73-maas-api-tls\") pod \"maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2\" (UID: \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:23.641711 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:23.641674    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:23.785916 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:23.785887    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"]
Jul 09 23:55:23.787260 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:55:23.787238    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb75bb75_ac24_4df3_ab47_bf05e6256f73.slice/crio-970df4e726c1c0b00f0bda943553b7c29eec80de644802f97174d19cb09af028 WatchSource:0}: Error finding container 970df4e726c1c0b00f0bda943553b7c29eec80de644802f97174d19cb09af028: Status 404 returned error can't find the container with id 970df4e726c1c0b00f0bda943553b7c29eec80de644802f97174d19cb09af028
Jul 09 23:55:24.757442 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:24.757400    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2" event={"ID":"fb75bb75-ac24-4df3-ab47-bf05e6256f73","Type":"ContainerStarted","Data":"3dfad3a9d8e4293f488643dbe786ff7a3522003ccb6f872fe59a7908c71d6c10"}
Jul 09 23:55:24.757862 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:24.757569    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:24.757862 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:24.757604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2" event={"ID":"fb75bb75-ac24-4df3-ab47-bf05e6256f73","Type":"ContainerStarted","Data":"970df4e726c1c0b00f0bda943553b7c29eec80de644802f97174d19cb09af028"}
Jul 09 23:55:24.772343 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:24.772299    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2" podStartSLOduration=2.358443335 podStartE2EDuration="2.77228331s" totalImagesPullingTime="413.839975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:55:24.771927655 +0000 UTC m=+1642.915858441" watchObservedRunningTime="2026-07-09 23:55:24.77228331 +0000 UTC m=+1642.916214095"
Jul 09 23:55:30.767883 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:30.767853    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:55:38.503561 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:38.502500    2580 patch_prober.go:28] interesting pod/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:57330->10.134.0.59:15021: read: connection reset by peer" start-of-body=
Jul 09 23:55:38.503561 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:38.502582    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" podUID="63610221-9416-4944-b620-fab17c28b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:57330->10.134.0.59:15021: read: connection reset by peer"
Jul 09 23:55:38.832479 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:38.832440    2580 generic.go:356] "Generic (PLEG): container finished" podID="63610221-9416-4944-b620-fab17c28b90b" containerID="ed7f4b0a2670d7a1e492e9dceedd04ccbe96b833e868f35de5a51b38a3a0ed03" exitCode=0
Jul 09 23:55:38.832711 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:38.832520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" event={"ID":"63610221-9416-4944-b620-fab17c28b90b","Type":"ContainerDied","Data":"ed7f4b0a2670d7a1e492e9dceedd04ccbe96b833e868f35de5a51b38a3a0ed03"}
Jul 09 23:55:38.832980 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:38.832963    2580 scope.go:117] "RemoveContainer" containerID="ed7f4b0a2670d7a1e492e9dceedd04ccbe96b833e868f35de5a51b38a3a0ed03"
Jul 09 23:55:38.834176 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:38.834144    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:55:38.834279 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:38.834228    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:55:38.834279 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:38.834272    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:55:39.843984 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:39.842997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" event={"ID":"63610221-9416-4944-b620-fab17c28b90b","Type":"ContainerStarted","Data":"e002d0ccda37b24785381f1c4636e1c98b9fd2aeffdaa9d56ce49331c2eec0fd"}
Jul 09 23:55:39.923050 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:39.923012    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:39.923270 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:39.923247    2580 patch_prober.go:28] interesting pod/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 09 23:55:39.923322 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:39.923294    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" podUID="63610221-9416-4944-b620-fab17c28b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 09 23:55:40.923434 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:40.923397    2580 patch_prober.go:28] interesting pod/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 09 23:55:40.924001 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:40.923458    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" podUID="63610221-9416-4944-b620-fab17c28b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 09 23:55:41.923411 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:41.923366    2580 patch_prober.go:28] interesting pod/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 09 23:55:41.923599 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:41.923436    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" podUID="63610221-9416-4944-b620-fab17c28b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 09 23:55:42.924201 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:42.924160    2580 patch_prober.go:28] interesting pod/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 09 23:55:42.924614 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:42.924218    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" podUID="63610221-9416-4944-b620-fab17c28b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 09 23:55:43.926620 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:43.926589    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:43.927107 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:43.927014    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:55:43.927718 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:55:43.927695    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:56:10.018355 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:10.018316    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"]
Jul 09 23:56:10.018853 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:10.018570    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2" podUID="fb75bb75-ac24-4df3-ab47-bf05e6256f73" containerName="maas-api" containerID="cri-o://3dfad3a9d8e4293f488643dbe786ff7a3522003ccb6f872fe59a7908c71d6c10" gracePeriod=30
Jul 09 23:56:10.264541 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:10.264516    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:56:10.332976 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:10.332873    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p584n\" (UniqueName: \"kubernetes.io/projected/fb75bb75-ac24-4df3-ab47-bf05e6256f73-kube-api-access-p584n\") pod \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\" (UID: \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\") "
Jul 09 23:56:10.332976 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:10.332944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb75bb75-ac24-4df3-ab47-bf05e6256f73-maas-api-tls\") pod \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\" (UID: \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\") "
Jul 09 23:56:10.333210 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:10.333086    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb75bb75-ac24-4df3-ab47-bf05e6256f73-openshift-service-ca\") pod \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\" (UID: \"fb75bb75-ac24-4df3-ab47-bf05e6256f73\") "
Jul 09 23:56:10.333414 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:10.333388    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb75bb75-ac24-4df3-ab47-bf05e6256f73-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb75bb75-ac24-4df3-ab47-bf05e6256f73" (UID: "fb75bb75-ac24-4df3-ab47-bf05e6256f73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 23:56:10.335060 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:10.335036    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb75bb75-ac24-4df3-ab47-bf05e6256f73-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb75bb75-ac24-4df3-ab47-bf05e6256f73" (UID: "fb75bb75-ac24-4df3-ab47-bf05e6256f73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 23:56:10.335390 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:10.335367    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb75bb75-ac24-4df3-ab47-bf05e6256f73-kube-api-access-p584n" (OuterVolumeSpecName: "kube-api-access-p584n") pod "fb75bb75-ac24-4df3-ab47-bf05e6256f73" (UID: "fb75bb75-ac24-4df3-ab47-bf05e6256f73"). InnerVolumeSpecName "kube-api-access-p584n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:56:10.434471 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:10.434437    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p584n\" (UniqueName: \"kubernetes.io/projected/fb75bb75-ac24-4df3-ab47-bf05e6256f73-kube-api-access-p584n\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:56:10.434471 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:10.434469    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb75bb75-ac24-4df3-ab47-bf05e6256f73-maas-api-tls\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:56:10.434685 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:10.434481    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb75bb75-ac24-4df3-ab47-bf05e6256f73-openshift-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:56:11.015623 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:11.015590    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb75bb75-ac24-4df3-ab47-bf05e6256f73" containerID="3dfad3a9d8e4293f488643dbe786ff7a3522003ccb6f872fe59a7908c71d6c10" exitCode=0
Jul 09 23:56:11.015852 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:11.015629    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"
Jul 09 23:56:11.015852 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:11.015679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2" event={"ID":"fb75bb75-ac24-4df3-ab47-bf05e6256f73","Type":"ContainerDied","Data":"3dfad3a9d8e4293f488643dbe786ff7a3522003ccb6f872fe59a7908c71d6c10"}
Jul 09 23:56:11.015852 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:11.015735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2" event={"ID":"fb75bb75-ac24-4df3-ab47-bf05e6256f73","Type":"ContainerDied","Data":"970df4e726c1c0b00f0bda943553b7c29eec80de644802f97174d19cb09af028"}
Jul 09 23:56:11.015852 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:11.015757    2580 scope.go:117] "RemoveContainer" containerID="3dfad3a9d8e4293f488643dbe786ff7a3522003ccb6f872fe59a7908c71d6c10"
Jul 09 23:56:11.024646 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:11.024445    2580 scope.go:117] "RemoveContainer" containerID="3dfad3a9d8e4293f488643dbe786ff7a3522003ccb6f872fe59a7908c71d6c10"
Jul 09 23:56:11.024889 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:56:11.024708    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfad3a9d8e4293f488643dbe786ff7a3522003ccb6f872fe59a7908c71d6c10\": container with ID starting with 3dfad3a9d8e4293f488643dbe786ff7a3522003ccb6f872fe59a7908c71d6c10 not found: ID does not exist" containerID="3dfad3a9d8e4293f488643dbe786ff7a3522003ccb6f872fe59a7908c71d6c10"
Jul 09 23:56:11.024889 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:11.024735    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfad3a9d8e4293f488643dbe786ff7a3522003ccb6f872fe59a7908c71d6c10"} err="failed to get container status \"3dfad3a9d8e4293f488643dbe786ff7a3522003ccb6f872fe59a7908c71d6c10\": rpc error: code = NotFound desc = could not find container \"3dfad3a9d8e4293f488643dbe786ff7a3522003ccb6f872fe59a7908c71d6c10\": container with ID starting with 3dfad3a9d8e4293f488643dbe786ff7a3522003ccb6f872fe59a7908c71d6c10 not found: ID does not exist"
Jul 09 23:56:11.033088 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:11.033061    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"]
Jul 09 23:56:11.035037 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:11.035018    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b0dea16-7bd7977fc9-9zfr2"]
Jul 09 23:56:12.418684 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:56:12.418635    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb75bb75-ac24-4df3-ab47-bf05e6256f73" path="/var/lib/kubelet/pods/fb75bb75-ac24-4df3-ab47-bf05e6256f73/volumes"
Jul 09 23:57:25.949184 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:25.949148    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"]
Jul 09 23:57:25.949838 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:25.949400    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" podUID="63610221-9416-4944-b620-fab17c28b90b" containerName="istio-proxy" containerID="cri-o://e002d0ccda37b24785381f1c4636e1c98b9fd2aeffdaa9d56ce49331c2eec0fd" gracePeriod=30
Jul 09 23:57:28.929728 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:28.929691    2580 patch_prober.go:28] interesting pod/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 09 23:57:28.930195 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:28.929753    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" podUID="63610221-9416-4944-b620-fab17c28b90b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 23:57:31.201004 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.200972    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:57:31.312258 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.312221    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f825k\" (UniqueName: \"kubernetes.io/projected/63610221-9416-4944-b620-fab17c28b90b-kube-api-access-f825k\") pod \"63610221-9416-4944-b620-fab17c28b90b\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") "
Jul 09 23:57:31.312475 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.312348    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-istio-envoy\") pod \"63610221-9416-4944-b620-fab17c28b90b\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") "
Jul 09 23:57:31.312475 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.312397    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-workload-certs\") pod \"63610221-9416-4944-b620-fab17c28b90b\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") "
Jul 09 23:57:31.312475 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.312429    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63610221-9416-4944-b620-fab17c28b90b-istiod-ca-cert\") pod \"63610221-9416-4944-b620-fab17c28b90b\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") "
Jul 09 23:57:31.312648 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.312500    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63610221-9416-4944-b620-fab17c28b90b-istio-podinfo\") pod \"63610221-9416-4944-b620-fab17c28b90b\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") "
Jul 09 23:57:31.312648 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.312563    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-credential-socket\") pod \"63610221-9416-4944-b620-fab17c28b90b\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") "
Jul 09 23:57:31.312648 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.312588    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-istio-data\") pod \"63610221-9416-4944-b620-fab17c28b90b\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") "
Jul 09 23:57:31.312648 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.312621    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-workload-socket\") pod \"63610221-9416-4944-b620-fab17c28b90b\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") "
Jul 09 23:57:31.312648 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.312617    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "63610221-9416-4944-b620-fab17c28b90b" (UID: "63610221-9416-4944-b620-fab17c28b90b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:57:31.312966 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.312693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63610221-9416-4944-b620-fab17c28b90b-istio-token\") pod \"63610221-9416-4944-b620-fab17c28b90b\" (UID: \"63610221-9416-4944-b620-fab17c28b90b\") "
Jul 09 23:57:31.312966 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.312792    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "63610221-9416-4944-b620-fab17c28b90b" (UID: "63610221-9416-4944-b620-fab17c28b90b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:57:31.312966 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.312821    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63610221-9416-4944-b620-fab17c28b90b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "63610221-9416-4944-b620-fab17c28b90b" (UID: "63610221-9416-4944-b620-fab17c28b90b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 23:57:31.312966 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.312934    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "63610221-9416-4944-b620-fab17c28b90b" (UID: "63610221-9416-4944-b620-fab17c28b90b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:57:31.313191 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.313000    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-credential-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:57:31.313191 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.313019    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-workload-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:57:31.313191 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.313033    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63610221-9416-4944-b620-fab17c28b90b-istiod-ca-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:57:31.313898 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.313864    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-istio-data" (OuterVolumeSpecName: "istio-data") pod "63610221-9416-4944-b620-fab17c28b90b" (UID: "63610221-9416-4944-b620-fab17c28b90b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:57:31.314923 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.314896    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63610221-9416-4944-b620-fab17c28b90b-istio-token" (OuterVolumeSpecName: "istio-token") pod "63610221-9416-4944-b620-fab17c28b90b" (UID: "63610221-9416-4944-b620-fab17c28b90b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:57:31.314923 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.314919    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "63610221-9416-4944-b620-fab17c28b90b" (UID: "63610221-9416-4944-b620-fab17c28b90b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 23:57:31.315102 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.314963    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63610221-9416-4944-b620-fab17c28b90b-kube-api-access-f825k" (OuterVolumeSpecName: "kube-api-access-f825k") pod "63610221-9416-4944-b620-fab17c28b90b" (UID: "63610221-9416-4944-b620-fab17c28b90b"). InnerVolumeSpecName "kube-api-access-f825k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 23:57:31.315102 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.315030    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63610221-9416-4944-b620-fab17c28b90b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "63610221-9416-4944-b620-fab17c28b90b" (UID: "63610221-9416-4944-b620-fab17c28b90b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 23:57:31.413384 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.413350    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-istio-data\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:57:31.413384 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.413381    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-workload-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:57:31.413384 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.413392    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63610221-9416-4944-b620-fab17c28b90b-istio-token\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:57:31.413625 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.413405    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f825k\" (UniqueName: \"kubernetes.io/projected/63610221-9416-4944-b620-fab17c28b90b-kube-api-access-f825k\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:57:31.413625 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.413417    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63610221-9416-4944-b620-fab17c28b90b-istio-envoy\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:57:31.413625 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.413430    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63610221-9416-4944-b620-fab17c28b90b-istio-podinfo\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 09 23:57:31.427921 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.427887    2580 generic.go:356] "Generic (PLEG): container finished" podID="63610221-9416-4944-b620-fab17c28b90b" containerID="e002d0ccda37b24785381f1c4636e1c98b9fd2aeffdaa9d56ce49331c2eec0fd" exitCode=0
Jul 09 23:57:31.428098 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.427948    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"
Jul 09 23:57:31.428098 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.427972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" event={"ID":"63610221-9416-4944-b620-fab17c28b90b","Type":"ContainerDied","Data":"e002d0ccda37b24785381f1c4636e1c98b9fd2aeffdaa9d56ce49331c2eec0fd"}
Jul 09 23:57:31.428098 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.428028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb" event={"ID":"63610221-9416-4944-b620-fab17c28b90b","Type":"ContainerDied","Data":"f7d8402c1d46fe2204033439798b8a32cd9536338ea8aa1bdf6fa99e523f26ad"}
Jul 09 23:57:31.428098 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.428046    2580 scope.go:117] "RemoveContainer" containerID="e002d0ccda37b24785381f1c4636e1c98b9fd2aeffdaa9d56ce49331c2eec0fd"
Jul 09 23:57:31.442973 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.442939    2580 scope.go:117] "RemoveContainer" containerID="ed7f4b0a2670d7a1e492e9dceedd04ccbe96b833e868f35de5a51b38a3a0ed03"
Jul 09 23:57:31.451909 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.451887    2580 scope.go:117] "RemoveContainer" containerID="e002d0ccda37b24785381f1c4636e1c98b9fd2aeffdaa9d56ce49331c2eec0fd"
Jul 09 23:57:31.452289 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:57:31.452253    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e002d0ccda37b24785381f1c4636e1c98b9fd2aeffdaa9d56ce49331c2eec0fd\": container with ID starting with e002d0ccda37b24785381f1c4636e1c98b9fd2aeffdaa9d56ce49331c2eec0fd not found: ID does not exist" containerID="e002d0ccda37b24785381f1c4636e1c98b9fd2aeffdaa9d56ce49331c2eec0fd"
Jul 09 23:57:31.452377 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.452290    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e002d0ccda37b24785381f1c4636e1c98b9fd2aeffdaa9d56ce49331c2eec0fd"} err="failed to get container status \"e002d0ccda37b24785381f1c4636e1c98b9fd2aeffdaa9d56ce49331c2eec0fd\": rpc error: code = NotFound desc = could not find container \"e002d0ccda37b24785381f1c4636e1c98b9fd2aeffdaa9d56ce49331c2eec0fd\": container with ID starting with e002d0ccda37b24785381f1c4636e1c98b9fd2aeffdaa9d56ce49331c2eec0fd not found: ID does not exist"
Jul 09 23:57:31.452377 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.452315    2580 scope.go:117] "RemoveContainer" containerID="ed7f4b0a2670d7a1e492e9dceedd04ccbe96b833e868f35de5a51b38a3a0ed03"
Jul 09 23:57:31.452860 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:57:31.452833    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7f4b0a2670d7a1e492e9dceedd04ccbe96b833e868f35de5a51b38a3a0ed03\": container with ID starting with ed7f4b0a2670d7a1e492e9dceedd04ccbe96b833e868f35de5a51b38a3a0ed03 not found: ID does not exist" containerID="ed7f4b0a2670d7a1e492e9dceedd04ccbe96b833e868f35de5a51b38a3a0ed03"
Jul 09 23:57:31.452985 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.452867    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7f4b0a2670d7a1e492e9dceedd04ccbe96b833e868f35de5a51b38a3a0ed03"} err="failed to get container status \"ed7f4b0a2670d7a1e492e9dceedd04ccbe96b833e868f35de5a51b38a3a0ed03\": rpc error: code = NotFound desc = could not find container \"ed7f4b0a2670d7a1e492e9dceedd04ccbe96b833e868f35de5a51b38a3a0ed03\": container with ID starting with ed7f4b0a2670d7a1e492e9dceedd04ccbe96b833e868f35de5a51b38a3a0ed03 not found: ID does not exist"
Jul 09 23:57:31.455151 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.455123    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"]
Jul 09 23:57:31.467300 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:31.467274    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7b0dea16-openshift-default-74867d6f9-dgzhb"]
Jul 09 23:57:32.420136 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:57:32.420105    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63610221-9416-4944-b620-fab17c28b90b" path="/var/lib/kubelet/pods/63610221-9416-4944-b620-fab17c28b90b/volumes"
Jul 09 23:58:29.586730 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.586641    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"]
Jul 09 23:58:29.587234 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.587039    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb75bb75-ac24-4df3-ab47-bf05e6256f73" containerName="maas-api"
Jul 09 23:58:29.587234 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.587052    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb75bb75-ac24-4df3-ab47-bf05e6256f73" containerName="maas-api"
Jul 09 23:58:29.587234 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.587060    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63610221-9416-4944-b620-fab17c28b90b" containerName="istio-proxy"
Jul 09 23:58:29.587234 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.587066    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="63610221-9416-4944-b620-fab17c28b90b" containerName="istio-proxy"
Jul 09 23:58:29.587234 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.587079    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63610221-9416-4944-b620-fab17c28b90b" containerName="istio-proxy"
Jul 09 23:58:29.587234 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.587085    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="63610221-9416-4944-b620-fab17c28b90b" containerName="istio-proxy"
Jul 09 23:58:29.587234 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.587155    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="63610221-9416-4944-b620-fab17c28b90b" containerName="istio-proxy"
Jul 09 23:58:29.587234 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.587163    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="63610221-9416-4944-b620-fab17c28b90b" containerName="istio-proxy"
Jul 09 23:58:29.587234 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.587170    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb75bb75-ac24-4df3-ab47-bf05e6256f73" containerName="maas-api"
Jul 09 23:58:29.590110 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.590089    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.592544 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.592524    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4e0be9-openshift-default-dockercfg-vgsr7\""
Jul 09 23:58:29.592727 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.592702    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 23:58:29.602834 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.602810    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"]
Jul 09 23:58:29.709904 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.709858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-workload-socket\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.709904 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.709906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-data\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.710122 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.709939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-workload-certs\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.710122 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.710006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de871a93-6645-4e7e-a846-2cd9e4c4f569-istiod-ca-cert\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.710122 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.710074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-token\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.710122 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.710106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mvgk\" (UniqueName: \"kubernetes.io/projected/de871a93-6645-4e7e-a846-2cd9e4c4f569-kube-api-access-6mvgk\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.710273 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.710134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-envoy\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.710273 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.710158    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-podinfo\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.710273 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.710204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-credential-socket\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.811624 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.811585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mvgk\" (UniqueName: \"kubernetes.io/projected/de871a93-6645-4e7e-a846-2cd9e4c4f569-kube-api-access-6mvgk\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.811624 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.811622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-envoy\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.811873 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.811638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-podinfo\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.811873 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.811695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-credential-socket\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.811873 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.811740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-workload-socket\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.811873 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.811761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-data\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.811873 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.811785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-workload-certs\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.811873 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.811807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de871a93-6645-4e7e-a846-2cd9e4c4f569-istiod-ca-cert\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.811873 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.811858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-token\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.812281 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.812137    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-workload-socket\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.812281 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.812257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-data\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.812388 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.812291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-workload-certs\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.812388 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.812358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-credential-socket\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.812695 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.812647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de871a93-6645-4e7e-a846-2cd9e4c4f569-istiod-ca-cert\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.813914 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.813897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-envoy\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.814207 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.814189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-podinfo\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.819192 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.819169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-token\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.819324 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.819200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mvgk\" (UniqueName: \"kubernetes.io/projected/de871a93-6645-4e7e-a846-2cd9e4c4f569-kube-api-access-6mvgk\") pod \"e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") " pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:29.902481 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:29.902385    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:30.044941 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:30.044915    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"]
Jul 09 23:58:30.047062 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:58:30.047032    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde871a93_6645_4e7e_a846_2cd9e4c4f569.slice/crio-e590741ba1b9f81549542e165b06f1e214c5200d9a8acbe4bfd0382c67668b49 WatchSource:0}: Error finding container e590741ba1b9f81549542e165b06f1e214c5200d9a8acbe4bfd0382c67668b49: Status 404 returned error can't find the container with id e590741ba1b9f81549542e165b06f1e214c5200d9a8acbe4bfd0382c67668b49
Jul 09 23:58:30.049222 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:30.049195    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:58:30.049290 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:30.049257    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:58:30.049290 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:30.049286    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:58:30.726895 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:30.726856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr" event={"ID":"de871a93-6645-4e7e-a846-2cd9e4c4f569","Type":"ContainerStarted","Data":"d941b129d5ee99fb84adcabaaf367f36aa9a147acb232535cc92bbeb5cffcb87"}
Jul 09 23:58:30.727320 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:30.726908    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr" event={"ID":"de871a93-6645-4e7e-a846-2cd9e4c4f569","Type":"ContainerStarted","Data":"e590741ba1b9f81549542e165b06f1e214c5200d9a8acbe4bfd0382c67668b49"}
Jul 09 23:58:30.747132 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:30.747057    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr" podStartSLOduration=1.747034948 podStartE2EDuration="1.747034948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:58:30.745342961 +0000 UTC m=+1828.889273748" watchObservedRunningTime="2026-07-09 23:58:30.747034948 +0000 UTC m=+1828.890965739"
Jul 09 23:58:30.903078 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:30.903041    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:31.907876 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:31.907840    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:32.720641 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.720608    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"]
Jul 09 23:58:32.724514 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.724488    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:32.726835 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.726807    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4e0be9-serving-cert\""
Jul 09 23:58:32.738580 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.738532    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"]
Jul 09 23:58:32.741165 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.741141    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:32.742244 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.742220    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:32.836420 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.836387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c78313da-f4cc-4e74-8306-9793c808515f-maas-api-tls\") pod \"maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8\" (UID: \"c78313da-f4cc-4e74-8306-9793c808515f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:32.836645 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.836462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78313da-f4cc-4e74-8306-9793c808515f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8\" (UID: \"c78313da-f4cc-4e74-8306-9793c808515f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:32.836645 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.836561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h74mt\" (UniqueName: \"kubernetes.io/projected/c78313da-f4cc-4e74-8306-9793c808515f-kube-api-access-h74mt\") pod \"maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8\" (UID: \"c78313da-f4cc-4e74-8306-9793c808515f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:32.937934 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.937893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c78313da-f4cc-4e74-8306-9793c808515f-maas-api-tls\") pod \"maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8\" (UID: \"c78313da-f4cc-4e74-8306-9793c808515f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:32.938393 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:58:32.938036    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-serving-cert: secret "maas-api-e2e-inf-a-4e0be9-serving-cert" not found
Jul 09 23:58:32.938393 ip-10-0-140-8 kubenswrapper[2580]: E0709 23:58:32.938127    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c78313da-f4cc-4e74-8306-9793c808515f-maas-api-tls podName:c78313da-f4cc-4e74-8306-9793c808515f nodeName:}" failed. No retries permitted until 2026-07-09 23:58:33.438106453 +0000 UTC m=+1831.582037240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c78313da-f4cc-4e74-8306-9793c808515f-maas-api-tls") pod "maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8" (UID: "c78313da-f4cc-4e74-8306-9793c808515f") : secret "maas-api-e2e-inf-a-4e0be9-serving-cert" not found
Jul 09 23:58:32.938393 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.938199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78313da-f4cc-4e74-8306-9793c808515f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8\" (UID: \"c78313da-f4cc-4e74-8306-9793c808515f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:32.938393 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.938247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74mt\" (UniqueName: \"kubernetes.io/projected/c78313da-f4cc-4e74-8306-9793c808515f-kube-api-access-h74mt\") pod \"maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8\" (UID: \"c78313da-f4cc-4e74-8306-9793c808515f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:32.938914 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.938893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78313da-f4cc-4e74-8306-9793c808515f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8\" (UID: \"c78313da-f4cc-4e74-8306-9793c808515f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:32.948222 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:32.948198    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h74mt\" (UniqueName: \"kubernetes.io/projected/c78313da-f4cc-4e74-8306-9793c808515f-kube-api-access-h74mt\") pod \"maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8\" (UID: \"c78313da-f4cc-4e74-8306-9793c808515f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:33.442570 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:33.442522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c78313da-f4cc-4e74-8306-9793c808515f-maas-api-tls\") pod \"maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8\" (UID: \"c78313da-f4cc-4e74-8306-9793c808515f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:33.445098 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:33.445064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c78313da-f4cc-4e74-8306-9793c808515f-maas-api-tls\") pod \"maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8\" (UID: \"c78313da-f4cc-4e74-8306-9793c808515f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:33.640474 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:33.640433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:33.783946 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:33.783915    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"]
Jul 09 23:58:33.785195 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:58:33.785169    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78313da_f4cc_4e74_8306_9793c808515f.slice/crio-4726c3bd6bd18fb545cc67cce866199fbd7be9b52cbec1d9b90a5c9383298df3 WatchSource:0}: Error finding container 4726c3bd6bd18fb545cc67cce866199fbd7be9b52cbec1d9b90a5c9383298df3: Status 404 returned error can't find the container with id 4726c3bd6bd18fb545cc67cce866199fbd7be9b52cbec1d9b90a5c9383298df3
Jul 09 23:58:33.786723 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:33.786703    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 23:58:34.753428 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:34.753386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8" event={"ID":"c78313da-f4cc-4e74-8306-9793c808515f","Type":"ContainerStarted","Data":"883a39124bf17de46f86d06781c6a4ddfef41c4ed163b955c4ce62bce9f1e356"}
Jul 09 23:58:34.753944 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:34.753454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8" event={"ID":"c78313da-f4cc-4e74-8306-9793c808515f","Type":"ContainerStarted","Data":"4726c3bd6bd18fb545cc67cce866199fbd7be9b52cbec1d9b90a5c9383298df3"}
Jul 09 23:58:34.753944 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:34.753591    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:34.769512 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:34.769450    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8" podStartSLOduration=2.308469305 podStartE2EDuration="2.76942986s" totalImagesPullingTime="460.960555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:58:34.767742402 +0000 UTC m=+1832.911673188" watchObservedRunningTime="2026-07-09 23:58:34.76942986 +0000 UTC m=+1832.913360647"
Jul 09 23:58:40.764066 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:40.764031    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 09 23:58:46.820222 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:46.820187    2580 generic.go:356] "Generic (PLEG): container finished" podID="de871a93-6645-4e7e-a846-2cd9e4c4f569" containerID="d941b129d5ee99fb84adcabaaf367f36aa9a147acb232535cc92bbeb5cffcb87" exitCode=0
Jul 09 23:58:46.820645 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:46.820228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr" event={"ID":"de871a93-6645-4e7e-a846-2cd9e4c4f569","Type":"ContainerDied","Data":"d941b129d5ee99fb84adcabaaf367f36aa9a147acb232535cc92bbeb5cffcb87"}
Jul 09 23:58:46.820645 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:46.820591    2580 scope.go:117] "RemoveContainer" containerID="d941b129d5ee99fb84adcabaaf367f36aa9a147acb232535cc92bbeb5cffcb87"
Jul 09 23:58:46.821535 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:46.821507    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:58:46.821598 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:46.821586    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:58:46.821642 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:46.821630    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:58:47.741431 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.741396    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:47.781077 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.781042    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"]
Jul 09 23:58:47.785599 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.785575    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.789535 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.789510    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4f4769-openshift-default-dockercfg-kxbgw\""
Jul 09 23:58:47.803785 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.803693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"]
Jul 09 23:58:47.836461 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.835781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr" event={"ID":"de871a93-6645-4e7e-a846-2cd9e4c4f569","Type":"ContainerStarted","Data":"86f8743850e13e60fd677e8af7990d268ce8c96d29f836000c5cd57d7f6f2d5c"}
Jul 09 23:58:47.862332 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.861842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bcce34-4560-459b-ac5f-f34d8f019042-istio-podinfo\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.862332 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.861994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-istio-envoy\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.862332 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.862043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-workload-certs\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.862332 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.862241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-istio-data\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.862710 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.862271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wln6h\" (UniqueName: \"kubernetes.io/projected/07bcce34-4560-459b-ac5f-f34d8f019042-kube-api-access-wln6h\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.862710 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.862482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bcce34-4560-459b-ac5f-f34d8f019042-istio-token\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.862710 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.862697    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-workload-socket\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.863212 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.863188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bcce34-4560-459b-ac5f-f34d8f019042-istiod-ca-cert\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.863328 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.863247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-credential-socket\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.903602 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.903565    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:47.904704 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.904682    2580 patch_prober.go:28] interesting pod/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 09 23:58:47.904766 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.904728    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr" podUID="de871a93-6645-4e7e-a846-2cd9e4c4f569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 09 23:58:47.964117 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.964074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bcce34-4560-459b-ac5f-f34d8f019042-istiod-ca-cert\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.964117 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.964119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-credential-socket\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.964333 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.964153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bcce34-4560-459b-ac5f-f34d8f019042-istio-podinfo\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.964333 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.964171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-istio-envoy\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.964333 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.964189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-workload-certs\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.964333 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.964203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-istio-data\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.964333 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.964224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wln6h\" (UniqueName: \"kubernetes.io/projected/07bcce34-4560-459b-ac5f-f34d8f019042-kube-api-access-wln6h\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.964333 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.964244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bcce34-4560-459b-ac5f-f34d8f019042-istio-token\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.964333 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.964269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-workload-socket\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.964618 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.964592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-workload-socket\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.964903 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.964883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-workload-certs\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.965266 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.965067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-istio-data\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.965266 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.965121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-credential-socket\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.965480 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.965463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bcce34-4560-459b-ac5f-f34d8f019042-istiod-ca-cert\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.966862 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.966833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-istio-envoy\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.967172 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.967151    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bcce34-4560-459b-ac5f-f34d8f019042-istio-podinfo\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.972616 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.972597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bcce34-4560-459b-ac5f-f34d8f019042-istio-token\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:47.972616 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:47.972613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wln6h\" (UniqueName: \"kubernetes.io/projected/07bcce34-4560-459b-ac5f-f34d8f019042-kube-api-access-wln6h\") pod \"e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") " pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:48.105671 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:48.105619    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:48.286441 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:48.286414    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"]
Jul 09 23:58:48.287988 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:58:48.287954    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bcce34_4560_459b_ac5f_f34d8f019042.slice/crio-050ca0629f5bddcb86fa3276d38f148068a087ad89c953d6d242c912940964f5 WatchSource:0}: Error finding container 050ca0629f5bddcb86fa3276d38f148068a087ad89c953d6d242c912940964f5: Status 404 returned error can't find the container with id 050ca0629f5bddcb86fa3276d38f148068a087ad89c953d6d242c912940964f5
Jul 09 23:58:48.290138 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:48.290111    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:58:48.290204 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:48.290171    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:58:48.290204 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:48.290198    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:58:48.847175 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:48.847067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" event={"ID":"07bcce34-4560-459b-ac5f-f34d8f019042","Type":"ContainerStarted","Data":"856b74501d838bca4d4d87b06a857b53b63827d89d5b4c09eac0b52480a94320"}
Jul 09 23:58:48.847175 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:48.847127    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" event={"ID":"07bcce34-4560-459b-ac5f-f34d8f019042","Type":"ContainerStarted","Data":"050ca0629f5bddcb86fa3276d38f148068a087ad89c953d6d242c912940964f5"}
Jul 09 23:58:48.868171 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:48.868104    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" podStartSLOduration=1.868080983 podStartE2EDuration="1.868080983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:58:48.864810269 +0000 UTC m=+1847.008741055" watchObservedRunningTime="2026-07-09 23:58:48.868080983 +0000 UTC m=+1847.012011770"
Jul 09 23:58:48.903407 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:48.903369    2580 patch_prober.go:28] interesting pod/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 09 23:58:48.903577 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:48.903424    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr" podUID="de871a93-6645-4e7e-a846-2cd9e4c4f569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 09 23:58:49.106097 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:49.106003    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:49.107711 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:49.107688    2580 patch_prober.go:28] interesting pod/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 09 23:58:49.107772 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:49.107737    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 09 23:58:49.903617 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:49.903578    2580 patch_prober.go:28] interesting pod/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 09 23:58:49.904004 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:49.903694    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr" podUID="de871a93-6645-4e7e-a846-2cd9e4c4f569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 09 23:58:50.110490 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:50.110452    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:50.868116 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:50.868018    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:50.869514 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:50.869490    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:58:50.903471 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:50.903435    2580 patch_prober.go:28] interesting pod/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 09 23:58:50.903670 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:50.903513    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr" podUID="de871a93-6645-4e7e-a846-2cd9e4c4f569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 09 23:58:51.907726 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:51.907493    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:51.909168 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:51.909143    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:58:51.914833 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:58:51.914803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 09 23:59:02.931235 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:02.931123    2580 generic.go:356] "Generic (PLEG): container finished" podID="07bcce34-4560-459b-ac5f-f34d8f019042" containerID="856b74501d838bca4d4d87b06a857b53b63827d89d5b4c09eac0b52480a94320" exitCode=0
Jul 09 23:59:02.931235 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:02.931203    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" event={"ID":"07bcce34-4560-459b-ac5f-f34d8f019042","Type":"ContainerDied","Data":"856b74501d838bca4d4d87b06a857b53b63827d89d5b4c09eac0b52480a94320"}
Jul 09 23:59:02.931851 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:02.931730    2580 scope.go:117] "RemoveContainer" containerID="856b74501d838bca4d4d87b06a857b53b63827d89d5b4c09eac0b52480a94320"
Jul 09 23:59:02.932930 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:02.932896    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:59:02.933029 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:02.932981    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:59:02.933074 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:02.933036    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 23:59:03.941576 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:03.941531    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" event={"ID":"07bcce34-4560-459b-ac5f-f34d8f019042","Type":"ContainerStarted","Data":"a9bfa8e4c39c54a1f7b62698118333175ff148ae736d06157dd8536d23c7d383"}
Jul 09 23:59:04.106270 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:04.106225    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:59:04.107620 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:04.107594    2580 patch_prober.go:28] interesting pod/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 09 23:59:04.107740 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:04.107645    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 09 23:59:05.107044 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:05.107005    2580 patch_prober.go:28] interesting pod/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 09 23:59:05.107471 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:05.107063    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 09 23:59:05.868464 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:05.868429    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:59:06.106875 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:06.106840    2580 patch_prober.go:28] interesting pod/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 09 23:59:06.107043 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:06.106899    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 09 23:59:07.107130 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:07.107088    2580 patch_prober.go:28] interesting pod/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 09 23:59:07.107538 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:07.107163    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 09 23:59:08.110840 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:08.110802    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:59:08.111985 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:08.111963    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 09 23:59:42.992839 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:42.992803    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"]
Jul 09 23:59:42.995233 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:42.995215    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:42.997574 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:42.997547    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4f4769\"/\"kube-root-ca.crt\""
Jul 09 23:59:42.998538 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:42.998493    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4f4769\"/\"default-dockercfg-fqslg\""
Jul 09 23:59:42.998676 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:42.998568    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4f4769\"/\"openshift-service-ca.crt\""
Jul 09 23:59:42.998676 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:42.998573    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4f4769\"/\"test-model-4f4769-kserve-self-signed-certs\""
Jul 09 23:59:43.005339 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.005314    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"]
Jul 09 23:59:43.058827 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.058791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd8b655a-88f0-41f2-8c9a-1c93c3061767-tls-certs\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.059020 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.058834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtrf5\" (UniqueName: \"kubernetes.io/projected/bd8b655a-88f0-41f2-8c9a-1c93c3061767-kube-api-access-xtrf5\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.059020 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.058898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-dshm\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.059020 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.058953    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-tmp-dir\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.059020 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.058985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-home\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.059020 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.059005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-model-cache\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.160219 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.160182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-dshm\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.160388 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.160230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-tmp-dir\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.160388 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.160265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-home\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.160388 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.160302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-model-cache\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.160388 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.160381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd8b655a-88f0-41f2-8c9a-1c93c3061767-tls-certs\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.160593 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.160409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtrf5\" (UniqueName: \"kubernetes.io/projected/bd8b655a-88f0-41f2-8c9a-1c93c3061767-kube-api-access-xtrf5\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.160787 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.160761    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-home\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.160881 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.160793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-tmp-dir\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.160881 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.160808    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-model-cache\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.162504 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.162481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-dshm\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.162999 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.162977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd8b655a-88f0-41f2-8c9a-1c93c3061767-tls-certs\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.167440 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.167417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtrf5\" (UniqueName: \"kubernetes.io/projected/bd8b655a-88f0-41f2-8c9a-1c93c3061767-kube-api-access-xtrf5\") pod \"test-model-4f4769-kserve-8466889d5-snn5z\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") " pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.304888 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.304854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:43.440098 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:43.440073    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"]
Jul 09 23:59:43.442293 ip-10-0-140-8 kubenswrapper[2580]: W0709 23:59:43.442263    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd8b655a_88f0_41f2_8c9a_1c93c3061767.slice/crio-37ad204a0146cc66f2102a9af70fcbed6c02ee48a0630bf411b296be69a4888c WatchSource:0}: Error finding container 37ad204a0146cc66f2102a9af70fcbed6c02ee48a0630bf411b296be69a4888c: Status 404 returned error can't find the container with id 37ad204a0146cc66f2102a9af70fcbed6c02ee48a0630bf411b296be69a4888c
Jul 09 23:59:44.152511 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:44.152471    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z" event={"ID":"bd8b655a-88f0-41f2-8c9a-1c93c3061767","Type":"ContainerStarted","Data":"37ad204a0146cc66f2102a9af70fcbed6c02ee48a0630bf411b296be69a4888c"}
Jul 09 23:59:45.158630 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:45.158593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z" event={"ID":"bd8b655a-88f0-41f2-8c9a-1c93c3061767","Type":"ContainerStarted","Data":"e4d83d72534b3744f28c74a67ce17ce0b1a1ee03d4797cfb1f367fc0bf34091d"}
Jul 09 23:59:45.173229 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:45.173178    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z" podStartSLOduration=2.06311039 podStartE2EDuration="3.17315991s" totalImagesPullingTime="1.11004952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:59:45.173088665 +0000 UTC m=+1903.317019454" watchObservedRunningTime="2026-07-09 23:59:45.17315991 +0000 UTC m=+1903.317090697"
Jul 09 23:59:53.305367 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:53.305332    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:53.305857 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:53.305638    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:53.319238 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:53.319212    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 09 23:59:54.219982 ip-10-0-140-8 kubenswrapper[2580]: I0709 23:59:54.219953    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 10 00:00:00.158324 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:00.158289    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz"]
Jul 10 00:00:00.160849 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:00.160825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz"
Jul 10 00:00:00.160995 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:00.160926    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq"]
Jul 10 00:00:00.163149 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:00.163125    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq"
Jul 10 00:00:00.163593 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:00.163576    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5vt95\""
Jul 10 00:00:00.177358 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:00.177329    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz"]
Jul 10 00:00:00.191882 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:00.191856    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq"]
Jul 10 00:00:00.198753 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:00.198730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz"
Jul 10 00:00:00.208047 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:00.208024    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq"
Jul 10 00:00:00.362296 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:00.362253    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz"]
Jul 10 00:00:00.365200 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:00:00.365168    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f3de04_d4b0_4622_a8c7_4934fd0ac540.slice/crio-9e4fac15f52527aed0b4efc394084a3ade0594e6a9a33c62e6ba5758b7ee2843 WatchSource:0}: Error finding container 9e4fac15f52527aed0b4efc394084a3ade0594e6a9a33c62e6ba5758b7ee2843: Status 404 returned error can't find the container with id 9e4fac15f52527aed0b4efc394084a3ade0594e6a9a33c62e6ba5758b7ee2843
Jul 10 00:00:00.585330 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:00.585303    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq"]
Jul 10 00:00:00.586339 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:00:00.586308    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738aabed_ade7_4d4b_bb43_9e4a2454786a.slice/crio-d36fd09ca9af75f544288d995c26083c63c69a409536822155a68b0ba6165b46 WatchSource:0}: Error finding container d36fd09ca9af75f544288d995c26083c63c69a409536822155a68b0ba6165b46: Status 404 returned error can't find the container with id d36fd09ca9af75f544288d995c26083c63c69a409536822155a68b0ba6165b46
Jul 10 00:00:01.251279 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:01.251172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz" event={"ID":"90f3de04-d4b0-4622-a8c7-4934fd0ac540","Type":"ContainerStarted","Data":"298d646b0882ac9495b566b8264e42b768c19179577731e42caeff1331affedf"}
Jul 10 00:00:01.251697 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:01.251278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz" event={"ID":"90f3de04-d4b0-4622-a8c7-4934fd0ac540","Type":"ContainerStarted","Data":"9e4fac15f52527aed0b4efc394084a3ade0594e6a9a33c62e6ba5758b7ee2843"}
Jul 10 00:00:01.252744 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:01.252721    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq" event={"ID":"738aabed-ade7-4d4b-bb43-9e4a2454786a","Type":"ContainerStarted","Data":"225b412786bf671737a39a513acf52f62cc22c22852e99659487c5462d095b68"}
Jul 10 00:00:01.252796 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:01.252763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq" event={"ID":"738aabed-ade7-4d4b-bb43-9e4a2454786a","Type":"ContainerStarted","Data":"d36fd09ca9af75f544288d995c26083c63c69a409536822155a68b0ba6165b46"}
Jul 10 00:00:01.268155 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:01.268105    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz" podStartSLOduration=1.268090895 podStartE2EDuration="1.268090895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:00:01.265795769 +0000 UTC m=+1919.409726558" watchObservedRunningTime="2026-07-10 00:00:01.268090895 +0000 UTC m=+1919.412021681"
Jul 10 00:00:01.282197 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:01.282146    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq" podStartSLOduration=1.28213183 podStartE2EDuration="1.28213183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:00:01.280204411 +0000 UTC m=+1919.424135200" watchObservedRunningTime="2026-07-10 00:00:01.28213183 +0000 UTC m=+1919.426062615"
Jul 10 00:00:15.119441 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:15.119401    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"]
Jul 10 00:00:15.951986 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:15.951949    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"]
Jul 10 00:00:15.952267 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:15.952201    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8" podUID="c78313da-f4cc-4e74-8306-9793c808515f" containerName="maas-api" containerID="cri-o://883a39124bf17de46f86d06781c6a4ddfef41c4ed163b955c4ce62bce9f1e356" gracePeriod=30
Jul 10 00:00:15.975682 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:15.975643    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz"]
Jul 10 00:00:15.975876 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:15.975842    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz" podUID="90f3de04-d4b0-4622-a8c7-4934fd0ac540" containerName="cleanup" containerID="cri-o://298d646b0882ac9495b566b8264e42b768c19179577731e42caeff1331affedf" gracePeriod=30
Jul 10 00:00:16.194864 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.194842    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 10 00:00:16.336638 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.336604    2580 generic.go:356] "Generic (PLEG): container finished" podID="c78313da-f4cc-4e74-8306-9793c808515f" containerID="883a39124bf17de46f86d06781c6a4ddfef41c4ed163b955c4ce62bce9f1e356" exitCode=0
Jul 10 00:00:16.336842 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.336696    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"
Jul 10 00:00:16.336842 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.336698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8" event={"ID":"c78313da-f4cc-4e74-8306-9793c808515f","Type":"ContainerDied","Data":"883a39124bf17de46f86d06781c6a4ddfef41c4ed163b955c4ce62bce9f1e356"}
Jul 10 00:00:16.336842 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.336765    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8" event={"ID":"c78313da-f4cc-4e74-8306-9793c808515f","Type":"ContainerDied","Data":"4726c3bd6bd18fb545cc67cce866199fbd7be9b52cbec1d9b90a5c9383298df3"}
Jul 10 00:00:16.336842 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.336795    2580 scope.go:117] "RemoveContainer" containerID="883a39124bf17de46f86d06781c6a4ddfef41c4ed163b955c4ce62bce9f1e356"
Jul 10 00:00:16.349168 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.349145    2580 scope.go:117] "RemoveContainer" containerID="883a39124bf17de46f86d06781c6a4ddfef41c4ed163b955c4ce62bce9f1e356"
Jul 10 00:00:16.349716 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:00:16.349696    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883a39124bf17de46f86d06781c6a4ddfef41c4ed163b955c4ce62bce9f1e356\": container with ID starting with 883a39124bf17de46f86d06781c6a4ddfef41c4ed163b955c4ce62bce9f1e356 not found: ID does not exist" containerID="883a39124bf17de46f86d06781c6a4ddfef41c4ed163b955c4ce62bce9f1e356"
Jul 10 00:00:16.349797 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.349725    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883a39124bf17de46f86d06781c6a4ddfef41c4ed163b955c4ce62bce9f1e356"} err="failed to get container status \"883a39124bf17de46f86d06781c6a4ddfef41c4ed163b955c4ce62bce9f1e356\": rpc error: code = NotFound desc = could not find container \"883a39124bf17de46f86d06781c6a4ddfef41c4ed163b955c4ce62bce9f1e356\": container with ID starting with 883a39124bf17de46f86d06781c6a4ddfef41c4ed163b955c4ce62bce9f1e356 not found: ID does not exist"
Jul 10 00:00:16.372095 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.372071    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c78313da-f4cc-4e74-8306-9793c808515f-maas-api-tls\") pod \"c78313da-f4cc-4e74-8306-9793c808515f\" (UID: \"c78313da-f4cc-4e74-8306-9793c808515f\") "
Jul 10 00:00:16.372186 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.372142    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h74mt\" (UniqueName: \"kubernetes.io/projected/c78313da-f4cc-4e74-8306-9793c808515f-kube-api-access-h74mt\") pod \"c78313da-f4cc-4e74-8306-9793c808515f\" (UID: \"c78313da-f4cc-4e74-8306-9793c808515f\") "
Jul 10 00:00:16.372186 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.372175    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78313da-f4cc-4e74-8306-9793c808515f-openshift-service-ca\") pod \"c78313da-f4cc-4e74-8306-9793c808515f\" (UID: \"c78313da-f4cc-4e74-8306-9793c808515f\") "
Jul 10 00:00:16.372541 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.372516    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78313da-f4cc-4e74-8306-9793c808515f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c78313da-f4cc-4e74-8306-9793c808515f" (UID: "c78313da-f4cc-4e74-8306-9793c808515f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 00:00:16.374143 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.374117    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c78313da-f4cc-4e74-8306-9793c808515f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c78313da-f4cc-4e74-8306-9793c808515f" (UID: "c78313da-f4cc-4e74-8306-9793c808515f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 00:00:16.374232 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.374175    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78313da-f4cc-4e74-8306-9793c808515f-kube-api-access-h74mt" (OuterVolumeSpecName: "kube-api-access-h74mt") pod "c78313da-f4cc-4e74-8306-9793c808515f" (UID: "c78313da-f4cc-4e74-8306-9793c808515f"). InnerVolumeSpecName "kube-api-access-h74mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:00:16.473086 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.473058    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h74mt\" (UniqueName: \"kubernetes.io/projected/c78313da-f4cc-4e74-8306-9793c808515f-kube-api-access-h74mt\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:00:16.473086 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.473087    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78313da-f4cc-4e74-8306-9793c808515f-openshift-service-ca\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:00:16.473292 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.473100    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c78313da-f4cc-4e74-8306-9793c808515f-maas-api-tls\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:00:16.652285 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.652215    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"]
Jul 10 00:00:16.655532 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:16.655502    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4e0be9-5687cf8b4d-b66p8"]
Jul 10 00:00:18.419911 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:18.419878    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c78313da-f4cc-4e74-8306-9793c808515f" path="/var/lib/kubelet/pods/c78313da-f4cc-4e74-8306-9793c808515f/volumes"
Jul 10 00:00:30.146161 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.146115    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z" podUID="bd8b655a-88f0-41f2-8c9a-1c93c3061767" containerName="main" containerID="cri-o://e4d83d72534b3744f28c74a67ce17ce0b1a1ee03d4797cfb1f367fc0bf34091d" gracePeriod=45
Jul 10 00:00:30.400988 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.400922    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 10 00:00:30.412262 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.412230    2580 generic.go:356] "Generic (PLEG): container finished" podID="bd8b655a-88f0-41f2-8c9a-1c93c3061767" containerID="e4d83d72534b3744f28c74a67ce17ce0b1a1ee03d4797cfb1f367fc0bf34091d" exitCode=0
Jul 10 00:00:30.412457 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.412267    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"
Jul 10 00:00:30.412457 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.412308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z" event={"ID":"bd8b655a-88f0-41f2-8c9a-1c93c3061767","Type":"ContainerDied","Data":"e4d83d72534b3744f28c74a67ce17ce0b1a1ee03d4797cfb1f367fc0bf34091d"}
Jul 10 00:00:30.412457 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.412358    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z" event={"ID":"bd8b655a-88f0-41f2-8c9a-1c93c3061767","Type":"ContainerDied","Data":"37ad204a0146cc66f2102a9af70fcbed6c02ee48a0630bf411b296be69a4888c"}
Jul 10 00:00:30.412457 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.412383    2580 scope.go:117] "RemoveContainer" containerID="e4d83d72534b3744f28c74a67ce17ce0b1a1ee03d4797cfb1f367fc0bf34091d"
Jul 10 00:00:30.422608 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.422580    2580 scope.go:117] "RemoveContainer" containerID="e4d83d72534b3744f28c74a67ce17ce0b1a1ee03d4797cfb1f367fc0bf34091d"
Jul 10 00:00:30.422954 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:00:30.422933    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d83d72534b3744f28c74a67ce17ce0b1a1ee03d4797cfb1f367fc0bf34091d\": container with ID starting with e4d83d72534b3744f28c74a67ce17ce0b1a1ee03d4797cfb1f367fc0bf34091d not found: ID does not exist" containerID="e4d83d72534b3744f28c74a67ce17ce0b1a1ee03d4797cfb1f367fc0bf34091d"
Jul 10 00:00:30.423054 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.422962    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d83d72534b3744f28c74a67ce17ce0b1a1ee03d4797cfb1f367fc0bf34091d"} err="failed to get container status \"e4d83d72534b3744f28c74a67ce17ce0b1a1ee03d4797cfb1f367fc0bf34091d\": rpc error: code = NotFound desc = could not find container \"e4d83d72534b3744f28c74a67ce17ce0b1a1ee03d4797cfb1f367fc0bf34091d\": container with ID starting with e4d83d72534b3744f28c74a67ce17ce0b1a1ee03d4797cfb1f367fc0bf34091d not found: ID does not exist"
Jul 10 00:00:30.489757 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.489722    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-dshm\") pod \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") "
Jul 10 00:00:30.489939 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.489780    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtrf5\" (UniqueName: \"kubernetes.io/projected/bd8b655a-88f0-41f2-8c9a-1c93c3061767-kube-api-access-xtrf5\") pod \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") "
Jul 10 00:00:30.489939 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.489799    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-tmp-dir\") pod \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") "
Jul 10 00:00:30.489939 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.489830    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd8b655a-88f0-41f2-8c9a-1c93c3061767-tls-certs\") pod \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") "
Jul 10 00:00:30.489939 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.489846    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-model-cache\") pod \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") "
Jul 10 00:00:30.489939 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.489878    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-home\") pod \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\" (UID: \"bd8b655a-88f0-41f2-8c9a-1c93c3061767\") "
Jul 10 00:00:30.490210 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.490109    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd8b655a-88f0-41f2-8c9a-1c93c3061767" (UID: "bd8b655a-88f0-41f2-8c9a-1c93c3061767"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:00:30.490210 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.490134    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd8b655a-88f0-41f2-8c9a-1c93c3061767" (UID: "bd8b655a-88f0-41f2-8c9a-1c93c3061767"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:00:30.490210 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.490183    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-home" (OuterVolumeSpecName: "home") pod "bd8b655a-88f0-41f2-8c9a-1c93c3061767" (UID: "bd8b655a-88f0-41f2-8c9a-1c93c3061767"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:00:30.492120 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.492095    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-dshm" (OuterVolumeSpecName: "dshm") pod "bd8b655a-88f0-41f2-8c9a-1c93c3061767" (UID: "bd8b655a-88f0-41f2-8c9a-1c93c3061767"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:00:30.492235 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.492138    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8b655a-88f0-41f2-8c9a-1c93c3061767-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd8b655a-88f0-41f2-8c9a-1c93c3061767" (UID: "bd8b655a-88f0-41f2-8c9a-1c93c3061767"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 00:00:30.492235 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.492173    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8b655a-88f0-41f2-8c9a-1c93c3061767-kube-api-access-xtrf5" (OuterVolumeSpecName: "kube-api-access-xtrf5") pod "bd8b655a-88f0-41f2-8c9a-1c93c3061767" (UID: "bd8b655a-88f0-41f2-8c9a-1c93c3061767"). InnerVolumeSpecName "kube-api-access-xtrf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:00:30.591193 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.591157    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-dshm\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:00:30.591193 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.591187    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtrf5\" (UniqueName: \"kubernetes.io/projected/bd8b655a-88f0-41f2-8c9a-1c93c3061767-kube-api-access-xtrf5\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:00:30.591193 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.591198    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-tmp-dir\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:00:30.591447 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.591208    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd8b655a-88f0-41f2-8c9a-1c93c3061767-tls-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:00:30.591447 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.591216    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-model-cache\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:00:30.591447 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.591225    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd8b655a-88f0-41f2-8c9a-1c93c3061767-home\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:00:30.733589 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.733558    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"]
Jul 10 00:00:30.736745 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:30.736719    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4f4769/test-model-4f4769-kserve-8466889d5-snn5z"]
Jul 10 00:00:32.419530 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:32.419487    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd8b655a-88f0-41f2-8c9a-1c93c3061767" path="/var/lib/kubelet/pods/bd8b655a-88f0-41f2-8c9a-1c93c3061767/volumes"
Jul 10 00:00:46.163048 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:46.163017    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz"
Jul 10 00:00:46.493857 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:46.493823    2580 generic.go:356] "Generic (PLEG): container finished" podID="90f3de04-d4b0-4622-a8c7-4934fd0ac540" containerID="298d646b0882ac9495b566b8264e42b768c19179577731e42caeff1331affedf" exitCode=137
Jul 10 00:00:46.494024 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:46.493865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz" event={"ID":"90f3de04-d4b0-4622-a8c7-4934fd0ac540","Type":"ContainerDied","Data":"298d646b0882ac9495b566b8264e42b768c19179577731e42caeff1331affedf"}
Jul 10 00:00:46.494024 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:46.493920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz" event={"ID":"90f3de04-d4b0-4622-a8c7-4934fd0ac540","Type":"ContainerDied","Data":"9e4fac15f52527aed0b4efc394084a3ade0594e6a9a33c62e6ba5758b7ee2843"}
Jul 10 00:00:46.494024 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:46.493948    2580 scope.go:117] "RemoveContainer" containerID="298d646b0882ac9495b566b8264e42b768c19179577731e42caeff1331affedf"
Jul 10 00:00:46.494024 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:46.493871    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz"
Jul 10 00:00:46.502767 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:46.502746    2580 scope.go:117] "RemoveContainer" containerID="298d646b0882ac9495b566b8264e42b768c19179577731e42caeff1331affedf"
Jul 10 00:00:46.503011 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:00:46.502993    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298d646b0882ac9495b566b8264e42b768c19179577731e42caeff1331affedf\": container with ID starting with 298d646b0882ac9495b566b8264e42b768c19179577731e42caeff1331affedf not found: ID does not exist" containerID="298d646b0882ac9495b566b8264e42b768c19179577731e42caeff1331affedf"
Jul 10 00:00:46.503062 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:46.503021    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298d646b0882ac9495b566b8264e42b768c19179577731e42caeff1331affedf"} err="failed to get container status \"298d646b0882ac9495b566b8264e42b768c19179577731e42caeff1331affedf\": rpc error: code = NotFound desc = could not find container \"298d646b0882ac9495b566b8264e42b768c19179577731e42caeff1331affedf\": container with ID starting with 298d646b0882ac9495b566b8264e42b768c19179577731e42caeff1331affedf not found: ID does not exist"
Jul 10 00:00:46.508695 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:46.508675    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz"]
Jul 10 00:00:46.514210 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:46.514190    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4e0be9-29727360-5b9wz"]
Jul 10 00:00:48.419295 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:48.419255    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90f3de04-d4b0-4622-a8c7-4934fd0ac540" path="/var/lib/kubelet/pods/90f3de04-d4b0-4622-a8c7-4934fd0ac540/volumes"
Jul 10 00:00:57.762286 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:57.762205    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"]
Jul 10 00:00:57.762690 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:00:57.762455    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr" podUID="de871a93-6645-4e7e-a846-2cd9e4c4f569" containerName="istio-proxy" containerID="cri-o://86f8743850e13e60fd677e8af7990d268ce8c96d29f836000c5cd57d7f6f2d5c" gracePeriod=30
Jul 10 00:01:00.009930 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:00.009894    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq"]
Jul 10 00:01:00.010320 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:00.010088    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq" podUID="738aabed-ade7-4d4b-bb43-9e4a2454786a" containerName="cleanup" containerID="cri-o://225b412786bf671737a39a513acf52f62cc22c22852e99659487c5462d095b68" gracePeriod=30
Jul 10 00:01:03.023946 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.023919    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 10 00:01:03.173505 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.173396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de871a93-6645-4e7e-a846-2cd9e4c4f569-istiod-ca-cert\") pod \"de871a93-6645-4e7e-a846-2cd9e4c4f569\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") "
Jul 10 00:01:03.173505 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.173462    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-data\") pod \"de871a93-6645-4e7e-a846-2cd9e4c4f569\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") "
Jul 10 00:01:03.173505 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.173494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-workload-certs\") pod \"de871a93-6645-4e7e-a846-2cd9e4c4f569\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") "
Jul 10 00:01:03.173822 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.173742    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-token\") pod \"de871a93-6645-4e7e-a846-2cd9e4c4f569\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") "
Jul 10 00:01:03.173822 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.173782    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-envoy\") pod \"de871a93-6645-4e7e-a846-2cd9e4c4f569\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") "
Jul 10 00:01:03.173928 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.173854    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-credential-socket\") pod \"de871a93-6645-4e7e-a846-2cd9e4c4f569\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") "
Jul 10 00:01:03.173928 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.173902    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-podinfo\") pod \"de871a93-6645-4e7e-a846-2cd9e4c4f569\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") "
Jul 10 00:01:03.173928 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.173925    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-workload-socket\") pod \"de871a93-6645-4e7e-a846-2cd9e4c4f569\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") "
Jul 10 00:01:03.174097 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.173951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mvgk\" (UniqueName: \"kubernetes.io/projected/de871a93-6645-4e7e-a846-2cd9e4c4f569-kube-api-access-6mvgk\") pod \"de871a93-6645-4e7e-a846-2cd9e4c4f569\" (UID: \"de871a93-6645-4e7e-a846-2cd9e4c4f569\") "
Jul 10 00:01:03.174097 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.173965    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de871a93-6645-4e7e-a846-2cd9e4c4f569" (UID: "de871a93-6645-4e7e-a846-2cd9e4c4f569"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:03.174097 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.174080    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de871a93-6645-4e7e-a846-2cd9e4c4f569-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de871a93-6645-4e7e-a846-2cd9e4c4f569" (UID: "de871a93-6645-4e7e-a846-2cd9e4c4f569"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 00:01:03.174327 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.174303    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de871a93-6645-4e7e-a846-2cd9e4c4f569" (UID: "de871a93-6645-4e7e-a846-2cd9e4c4f569"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:03.174408 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.174309    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de871a93-6645-4e7e-a846-2cd9e4c4f569-istiod-ca-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:03.174408 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.174348    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-workload-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:03.174495 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.174466    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de871a93-6645-4e7e-a846-2cd9e4c4f569" (UID: "de871a93-6645-4e7e-a846-2cd9e4c4f569"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:03.174787 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.174750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-data" (OuterVolumeSpecName: "istio-data") pod "de871a93-6645-4e7e-a846-2cd9e4c4f569" (UID: "de871a93-6645-4e7e-a846-2cd9e4c4f569"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:03.176117 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.176091    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de871a93-6645-4e7e-a846-2cd9e4c4f569" (UID: "de871a93-6645-4e7e-a846-2cd9e4c4f569"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:03.176210 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.176167    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de871a93-6645-4e7e-a846-2cd9e4c4f569" (UID: "de871a93-6645-4e7e-a846-2cd9e4c4f569"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 10 00:01:03.176253 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.176203    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-token" (OuterVolumeSpecName: "istio-token") pod "de871a93-6645-4e7e-a846-2cd9e4c4f569" (UID: "de871a93-6645-4e7e-a846-2cd9e4c4f569"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:01:03.176417 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.176403    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de871a93-6645-4e7e-a846-2cd9e4c4f569-kube-api-access-6mvgk" (OuterVolumeSpecName: "kube-api-access-6mvgk") pod "de871a93-6645-4e7e-a846-2cd9e4c4f569" (UID: "de871a93-6645-4e7e-a846-2cd9e4c4f569"). InnerVolumeSpecName "kube-api-access-6mvgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:01:03.275196 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.275152    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-podinfo\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:03.275196 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.275190    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-workload-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:03.275196 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.275199    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mvgk\" (UniqueName: \"kubernetes.io/projected/de871a93-6645-4e7e-a846-2cd9e4c4f569-kube-api-access-6mvgk\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:03.275445 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.275211    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-data\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:03.275445 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.275220    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-token\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:03.275445 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.275227    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-istio-envoy\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:03.275445 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.275234    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de871a93-6645-4e7e-a846-2cd9e4c4f569-credential-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:03.587256 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.587203    2580 generic.go:356] "Generic (PLEG): container finished" podID="de871a93-6645-4e7e-a846-2cd9e4c4f569" containerID="86f8743850e13e60fd677e8af7990d268ce8c96d29f836000c5cd57d7f6f2d5c" exitCode=0
Jul 10 00:01:03.587450 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.587292    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"
Jul 10 00:01:03.587450 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.587292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr" event={"ID":"de871a93-6645-4e7e-a846-2cd9e4c4f569","Type":"ContainerDied","Data":"86f8743850e13e60fd677e8af7990d268ce8c96d29f836000c5cd57d7f6f2d5c"}
Jul 10 00:01:03.587450 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.587362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr" event={"ID":"de871a93-6645-4e7e-a846-2cd9e4c4f569","Type":"ContainerDied","Data":"e590741ba1b9f81549542e165b06f1e214c5200d9a8acbe4bfd0382c67668b49"}
Jul 10 00:01:03.587450 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.587394    2580 scope.go:117] "RemoveContainer" containerID="86f8743850e13e60fd677e8af7990d268ce8c96d29f836000c5cd57d7f6f2d5c"
Jul 10 00:01:03.597950 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.597933    2580 scope.go:117] "RemoveContainer" containerID="d941b129d5ee99fb84adcabaaf367f36aa9a147acb232535cc92bbeb5cffcb87"
Jul 10 00:01:03.606022 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.606003    2580 scope.go:117] "RemoveContainer" containerID="86f8743850e13e60fd677e8af7990d268ce8c96d29f836000c5cd57d7f6f2d5c"
Jul 10 00:01:03.606283 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:01:03.606255    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f8743850e13e60fd677e8af7990d268ce8c96d29f836000c5cd57d7f6f2d5c\": container with ID starting with 86f8743850e13e60fd677e8af7990d268ce8c96d29f836000c5cd57d7f6f2d5c not found: ID does not exist" containerID="86f8743850e13e60fd677e8af7990d268ce8c96d29f836000c5cd57d7f6f2d5c"
Jul 10 00:01:03.606390 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.606282    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f8743850e13e60fd677e8af7990d268ce8c96d29f836000c5cd57d7f6f2d5c"} err="failed to get container status \"86f8743850e13e60fd677e8af7990d268ce8c96d29f836000c5cd57d7f6f2d5c\": rpc error: code = NotFound desc = could not find container \"86f8743850e13e60fd677e8af7990d268ce8c96d29f836000c5cd57d7f6f2d5c\": container with ID starting with 86f8743850e13e60fd677e8af7990d268ce8c96d29f836000c5cd57d7f6f2d5c not found: ID does not exist"
Jul 10 00:01:03.606390 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.606300    2580 scope.go:117] "RemoveContainer" containerID="d941b129d5ee99fb84adcabaaf367f36aa9a147acb232535cc92bbeb5cffcb87"
Jul 10 00:01:03.606543 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:01:03.606527    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d941b129d5ee99fb84adcabaaf367f36aa9a147acb232535cc92bbeb5cffcb87\": container with ID starting with d941b129d5ee99fb84adcabaaf367f36aa9a147acb232535cc92bbeb5cffcb87 not found: ID does not exist" containerID="d941b129d5ee99fb84adcabaaf367f36aa9a147acb232535cc92bbeb5cffcb87"
Jul 10 00:01:03.606605 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.606550    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d941b129d5ee99fb84adcabaaf367f36aa9a147acb232535cc92bbeb5cffcb87"} err="failed to get container status \"d941b129d5ee99fb84adcabaaf367f36aa9a147acb232535cc92bbeb5cffcb87\": rpc error: code = NotFound desc = could not find container \"d941b129d5ee99fb84adcabaaf367f36aa9a147acb232535cc92bbeb5cffcb87\": container with ID starting with d941b129d5ee99fb84adcabaaf367f36aa9a147acb232535cc92bbeb5cffcb87 not found: ID does not exist"
Jul 10 00:01:03.609706 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.609683    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"]
Jul 10 00:01:03.614298 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:03.614275    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4e0be9-openshift-default-8576984df5-j5znr"]
Jul 10 00:01:04.418687 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:04.418635    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de871a93-6645-4e7e-a846-2cd9e4c4f569" path="/var/lib/kubelet/pods/de871a93-6645-4e7e-a846-2cd9e4c4f569/volumes"
Jul 10 00:01:04.524903 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:04.524872    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"]
Jul 10 00:01:04.525106 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:04.525080    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" containerName="istio-proxy" containerID="cri-o://a9bfa8e4c39c54a1f7b62698118333175ff148ae736d06157dd8536d23c7d383" gracePeriod=30
Jul 10 00:01:05.319315 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.319284    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"]
Jul 10 00:01:05.319849 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.319827    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c78313da-f4cc-4e74-8306-9793c808515f" containerName="maas-api"
Jul 10 00:01:05.319849 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.319848    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c78313da-f4cc-4e74-8306-9793c808515f" containerName="maas-api"
Jul 10 00:01:05.320046 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.319881    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de871a93-6645-4e7e-a846-2cd9e4c4f569" containerName="istio-proxy"
Jul 10 00:01:05.320046 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.319890    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="de871a93-6645-4e7e-a846-2cd9e4c4f569" containerName="istio-proxy"
Jul 10 00:01:05.320046 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.319905    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de871a93-6645-4e7e-a846-2cd9e4c4f569" containerName="istio-proxy"
Jul 10 00:01:05.320046 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.319914    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="de871a93-6645-4e7e-a846-2cd9e4c4f569" containerName="istio-proxy"
Jul 10 00:01:05.320046 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.319923    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f3de04-d4b0-4622-a8c7-4934fd0ac540" containerName="cleanup"
Jul 10 00:01:05.320046 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.319931    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f3de04-d4b0-4622-a8c7-4934fd0ac540" containerName="cleanup"
Jul 10 00:01:05.320046 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.319967    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8b655a-88f0-41f2-8c9a-1c93c3061767" containerName="main"
Jul 10 00:01:05.320046 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.319976    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8b655a-88f0-41f2-8c9a-1c93c3061767" containerName="main"
Jul 10 00:01:05.320388 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.320057    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="de871a93-6645-4e7e-a846-2cd9e4c4f569" containerName="istio-proxy"
Jul 10 00:01:05.320388 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.320069    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="de871a93-6645-4e7e-a846-2cd9e4c4f569" containerName="istio-proxy"
Jul 10 00:01:05.320388 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.320078    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f3de04-d4b0-4622-a8c7-4934fd0ac540" containerName="cleanup"
Jul 10 00:01:05.320388 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.320087    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c78313da-f4cc-4e74-8306-9793c808515f" containerName="maas-api"
Jul 10 00:01:05.320388 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.320098    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8b655a-88f0-41f2-8c9a-1c93c3061767" containerName="main"
Jul 10 00:01:05.325149 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.325129    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.327443 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.327425    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-04eddb-openshift-default-dockercfg-2csgz\""
Jul 10 00:01:05.333567 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.333437    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"]
Jul 10 00:01:05.494682 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.494621    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-workload-socket\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.494682 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.494674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-credential-socket\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.495197 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.494694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-podinfo\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.495197 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.494764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-envoy\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.495197 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.494812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-workload-certs\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.495197 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.494833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-token\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.495197 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.494881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-data\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.495197 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.494931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz55z\" (UniqueName: \"kubernetes.io/projected/a1c36835-ac57-4118-bc68-cd681cd32f60-kube-api-access-fz55z\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.495197 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.494962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c36835-ac57-4118-bc68-cd681cd32f60-istiod-ca-cert\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.595984 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.595905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz55z\" (UniqueName: \"kubernetes.io/projected/a1c36835-ac57-4118-bc68-cd681cd32f60-kube-api-access-fz55z\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.595984 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.595949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c36835-ac57-4118-bc68-cd681cd32f60-istiod-ca-cert\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.596162 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.595986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-workload-socket\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.596162 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.596022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-credential-socket\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.596162 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.596147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-podinfo\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.596305 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.596208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-envoy\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.596305 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.596269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-workload-certs\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.596414 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.596304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-token\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.596414 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.596334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-data\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.596414 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.596390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-workload-socket\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.596748 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.596620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-workload-certs\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.596748 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.596734    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-credential-socket\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.596927 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.596753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-data\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.596927 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.596812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c36835-ac57-4118-bc68-cd681cd32f60-istiod-ca-cert\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.598709 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.598685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-envoy\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.598843 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.598711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-podinfo\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.603603 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.603570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-token\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.603756 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.603698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz55z\" (UniqueName: \"kubernetes.io/projected/a1c36835-ac57-4118-bc68-cd681cd32f60-kube-api-access-fz55z\") pod \"e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") " pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.639324 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.639294    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:05.789406 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.789382    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"]
Jul 10 00:01:05.791567 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:01:05.791543    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c36835_ac57_4118_bc68_cd681cd32f60.slice/crio-6ee7b541d70e45f66c7157caab24ef31aa514ce72a5bba0b4dff4600e065ac3e WatchSource:0}: Error finding container 6ee7b541d70e45f66c7157caab24ef31aa514ce72a5bba0b4dff4600e065ac3e: Status 404 returned error can't find the container with id 6ee7b541d70e45f66c7157caab24ef31aa514ce72a5bba0b4dff4600e065ac3e
Jul 10 00:01:05.794111 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.794079    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:05.794254 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.794145    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:05.794254 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:05.794179    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:06.606157 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:06.606122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b" event={"ID":"a1c36835-ac57-4118-bc68-cd681cd32f60","Type":"ContainerStarted","Data":"4cb61f6dddcc0354112b27ee10993fb0685d371c55c7848431918dfc503f1b5e"}
Jul 10 00:01:06.606627 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:06.606191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b" event={"ID":"a1c36835-ac57-4118-bc68-cd681cd32f60","Type":"ContainerStarted","Data":"6ee7b541d70e45f66c7157caab24ef31aa514ce72a5bba0b4dff4600e065ac3e"}
Jul 10 00:01:06.625909 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:06.625862    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b" podStartSLOduration=1.625847201 podStartE2EDuration="1.625847201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:01:06.623375876 +0000 UTC m=+1984.767306661" watchObservedRunningTime="2026-07-10 00:01:06.625847201 +0000 UTC m=+1984.769777987"
Jul 10 00:01:06.640082 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:06.640053    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:06.645142 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:06.645116    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:07.611147 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:07.611117    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:07.612112 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:07.612093    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:08.113429 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:08.113392    2580 patch_prober.go:28] interesting pod/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 10 00:01:08.113584 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:08.113454    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 00:01:09.779858 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.779829    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 10 00:01:09.936062 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.935970    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-workload-certs\") pod \"07bcce34-4560-459b-ac5f-f34d8f019042\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") "
Jul 10 00:01:09.936062 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.936008    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bcce34-4560-459b-ac5f-f34d8f019042-istiod-ca-cert\") pod \"07bcce34-4560-459b-ac5f-f34d8f019042\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") "
Jul 10 00:01:09.936062 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.936034    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-workload-socket\") pod \"07bcce34-4560-459b-ac5f-f34d8f019042\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") "
Jul 10 00:01:09.936062 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.936061    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bcce34-4560-459b-ac5f-f34d8f019042-istio-token\") pod \"07bcce34-4560-459b-ac5f-f34d8f019042\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") "
Jul 10 00:01:09.936385 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.936083    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-credential-socket\") pod \"07bcce34-4560-459b-ac5f-f34d8f019042\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") "
Jul 10 00:01:09.936385 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.936142    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-istio-data\") pod \"07bcce34-4560-459b-ac5f-f34d8f019042\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") "
Jul 10 00:01:09.936385 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.936173    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bcce34-4560-459b-ac5f-f34d8f019042-istio-podinfo\") pod \"07bcce34-4560-459b-ac5f-f34d8f019042\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") "
Jul 10 00:01:09.936385 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.936210    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-istio-envoy\") pod \"07bcce34-4560-459b-ac5f-f34d8f019042\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") "
Jul 10 00:01:09.936385 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.936269    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wln6h\" (UniqueName: \"kubernetes.io/projected/07bcce34-4560-459b-ac5f-f34d8f019042-kube-api-access-wln6h\") pod \"07bcce34-4560-459b-ac5f-f34d8f019042\" (UID: \"07bcce34-4560-459b-ac5f-f34d8f019042\") "
Jul 10 00:01:09.936385 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.936335    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "07bcce34-4560-459b-ac5f-f34d8f019042" (UID: "07bcce34-4560-459b-ac5f-f34d8f019042"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:09.936385 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.936349    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "07bcce34-4560-459b-ac5f-f34d8f019042" (UID: "07bcce34-4560-459b-ac5f-f34d8f019042"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:09.936764 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.936540    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-workload-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:09.936764 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.936559    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-workload-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:09.936764 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.936687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "07bcce34-4560-459b-ac5f-f34d8f019042" (UID: "07bcce34-4560-459b-ac5f-f34d8f019042"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:09.937124 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.937097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07bcce34-4560-459b-ac5f-f34d8f019042-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "07bcce34-4560-459b-ac5f-f34d8f019042" (UID: "07bcce34-4560-459b-ac5f-f34d8f019042"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 00:01:09.937304 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.937193    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-istio-data" (OuterVolumeSpecName: "istio-data") pod "07bcce34-4560-459b-ac5f-f34d8f019042" (UID: "07bcce34-4560-459b-ac5f-f34d8f019042"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:09.938727 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.938703    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07bcce34-4560-459b-ac5f-f34d8f019042-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "07bcce34-4560-459b-ac5f-f34d8f019042" (UID: "07bcce34-4560-459b-ac5f-f34d8f019042"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 10 00:01:09.938897 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.938871    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "07bcce34-4560-459b-ac5f-f34d8f019042" (UID: "07bcce34-4560-459b-ac5f-f34d8f019042"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:09.938897 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.938891    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bcce34-4560-459b-ac5f-f34d8f019042-istio-token" (OuterVolumeSpecName: "istio-token") pod "07bcce34-4560-459b-ac5f-f34d8f019042" (UID: "07bcce34-4560-459b-ac5f-f34d8f019042"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:01:09.939079 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:09.939057    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bcce34-4560-459b-ac5f-f34d8f019042-kube-api-access-wln6h" (OuterVolumeSpecName: "kube-api-access-wln6h") pod "07bcce34-4560-459b-ac5f-f34d8f019042" (UID: "07bcce34-4560-459b-ac5f-f34d8f019042"). InnerVolumeSpecName "kube-api-access-wln6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:01:10.037448 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.037415    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bcce34-4560-459b-ac5f-f34d8f019042-istio-podinfo\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:10.037448 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.037444    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-istio-envoy\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:10.037448 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.037455    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wln6h\" (UniqueName: \"kubernetes.io/projected/07bcce34-4560-459b-ac5f-f34d8f019042-kube-api-access-wln6h\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:10.037694 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.037466    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bcce34-4560-459b-ac5f-f34d8f019042-istiod-ca-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:10.037694 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.037476    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bcce34-4560-459b-ac5f-f34d8f019042-istio-token\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:10.037694 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.037483    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-credential-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:10.037694 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.037491    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bcce34-4560-459b-ac5f-f34d8f019042-istio-data\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:10.628366 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.628329    2580 generic.go:356] "Generic (PLEG): container finished" podID="07bcce34-4560-459b-ac5f-f34d8f019042" containerID="a9bfa8e4c39c54a1f7b62698118333175ff148ae736d06157dd8536d23c7d383" exitCode=0
Jul 10 00:01:10.628539 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.628387    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"
Jul 10 00:01:10.628539 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.628418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" event={"ID":"07bcce34-4560-459b-ac5f-f34d8f019042","Type":"ContainerDied","Data":"a9bfa8e4c39c54a1f7b62698118333175ff148ae736d06157dd8536d23c7d383"}
Jul 10 00:01:10.628539 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.628485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6" event={"ID":"07bcce34-4560-459b-ac5f-f34d8f019042","Type":"ContainerDied","Data":"050ca0629f5bddcb86fa3276d38f148068a087ad89c953d6d242c912940964f5"}
Jul 10 00:01:10.628539 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.628516    2580 scope.go:117] "RemoveContainer" containerID="a9bfa8e4c39c54a1f7b62698118333175ff148ae736d06157dd8536d23c7d383"
Jul 10 00:01:10.638600 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.638581    2580 scope.go:117] "RemoveContainer" containerID="856b74501d838bca4d4d87b06a857b53b63827d89d5b4c09eac0b52480a94320"
Jul 10 00:01:10.645974 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.645941    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"]
Jul 10 00:01:10.647931 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.647790    2580 scope.go:117] "RemoveContainer" containerID="a9bfa8e4c39c54a1f7b62698118333175ff148ae736d06157dd8536d23c7d383"
Jul 10 00:01:10.648082 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:01:10.648049    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9bfa8e4c39c54a1f7b62698118333175ff148ae736d06157dd8536d23c7d383\": container with ID starting with a9bfa8e4c39c54a1f7b62698118333175ff148ae736d06157dd8536d23c7d383 not found: ID does not exist" containerID="a9bfa8e4c39c54a1f7b62698118333175ff148ae736d06157dd8536d23c7d383"
Jul 10 00:01:10.648258 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.648076    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9bfa8e4c39c54a1f7b62698118333175ff148ae736d06157dd8536d23c7d383"} err="failed to get container status \"a9bfa8e4c39c54a1f7b62698118333175ff148ae736d06157dd8536d23c7d383\": rpc error: code = NotFound desc = could not find container \"a9bfa8e4c39c54a1f7b62698118333175ff148ae736d06157dd8536d23c7d383\": container with ID starting with a9bfa8e4c39c54a1f7b62698118333175ff148ae736d06157dd8536d23c7d383 not found: ID does not exist"
Jul 10 00:01:10.648258 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.648094    2580 scope.go:117] "RemoveContainer" containerID="856b74501d838bca4d4d87b06a857b53b63827d89d5b4c09eac0b52480a94320"
Jul 10 00:01:10.648421 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:01:10.648377    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856b74501d838bca4d4d87b06a857b53b63827d89d5b4c09eac0b52480a94320\": container with ID starting with 856b74501d838bca4d4d87b06a857b53b63827d89d5b4c09eac0b52480a94320 not found: ID does not exist" containerID="856b74501d838bca4d4d87b06a857b53b63827d89d5b4c09eac0b52480a94320"
Jul 10 00:01:10.648421 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.648402    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856b74501d838bca4d4d87b06a857b53b63827d89d5b4c09eac0b52480a94320"} err="failed to get container status \"856b74501d838bca4d4d87b06a857b53b63827d89d5b4c09eac0b52480a94320\": rpc error: code = NotFound desc = could not find container \"856b74501d838bca4d4d87b06a857b53b63827d89d5b4c09eac0b52480a94320\": container with ID starting with 856b74501d838bca4d4d87b06a857b53b63827d89d5b4c09eac0b52480a94320 not found: ID does not exist"
Jul 10 00:01:10.650748 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:10.650729    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4f4769-openshift-default-6564cb5875-wk7q6"]
Jul 10 00:01:12.419063 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:12.419022    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" path="/var/lib/kubelet/pods/07bcce34-4560-459b-ac5f-f34d8f019042/volumes"
Jul 10 00:01:13.712194 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.712157    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"]
Jul 10 00:01:13.712810 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.712785    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" containerName="istio-proxy"
Jul 10 00:01:13.712993 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.712976    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" containerName="istio-proxy"
Jul 10 00:01:13.713087 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.713068    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" containerName="istio-proxy"
Jul 10 00:01:13.713153 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.713089    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" containerName="istio-proxy"
Jul 10 00:01:13.713212 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.713194    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" containerName="istio-proxy"
Jul 10 00:01:13.713212 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.713208    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07bcce34-4560-459b-ac5f-f34d8f019042" containerName="istio-proxy"
Jul 10 00:01:13.717780 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.717754    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.720293 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.720271    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9ed59f-openshift-default-dockercfg-gsjgn\""
Jul 10 00:01:13.726928 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.726800    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"]
Jul 10 00:01:13.764399 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.764371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh72s\" (UniqueName: \"kubernetes.io/projected/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-kube-api-access-qh72s\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.764538 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.764427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-envoy\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.764538 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.764462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-token\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.764538 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.764490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istiod-ca-cert\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.764538 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.764514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-workload-socket\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.764697 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.764554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-podinfo\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.764697 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.764576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-data\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.764697 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.764600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-credential-socket\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.764697 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.764619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-workload-certs\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.865151 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.865095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh72s\" (UniqueName: \"kubernetes.io/projected/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-kube-api-access-qh72s\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.865349 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.865237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-envoy\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.865349 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.865254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-token\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.865349 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.865272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istiod-ca-cert\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.865349 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.865290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-workload-socket\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.865349 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.865331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-podinfo\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.865621 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.865354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-data\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.865621 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.865379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-credential-socket\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.865621 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.865408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-workload-certs\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.865883 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.865856    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-workload-socket\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.865943 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.865868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-workload-certs\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.866157 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.866127    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-credential-socket\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.866157 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.866153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istiod-ca-cert\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.866356 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.866184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-data\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.867684 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.867647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-envoy\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.868330 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.868312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-podinfo\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.873071 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.873041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-token\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:13.873174 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:13.873146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh72s\" (UniqueName: \"kubernetes.io/projected/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-kube-api-access-qh72s\") pod \"e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") " pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:14.031961 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:14.031926    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:14.181677 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:14.181636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"]
Jul 10 00:01:14.183359 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:01:14.183324    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe74446_ebb5_4f10_8bd2_6bd7df25eceb.slice/crio-7ff53241f667a3089dddddae7bd945fd8c2998e7d447eeb1ec097c5cf8431543 WatchSource:0}: Error finding container 7ff53241f667a3089dddddae7bd945fd8c2998e7d447eeb1ec097c5cf8431543: Status 404 returned error can't find the container with id 7ff53241f667a3089dddddae7bd945fd8c2998e7d447eeb1ec097c5cf8431543
Jul 10 00:01:14.185473 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:14.185438    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:14.185583 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:14.185517    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:14.185583 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:14.185561    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:14.653806 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:14.653773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b" event={"ID":"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb","Type":"ContainerStarted","Data":"4836af8814f0e44dff72907c061d7bea5a49a22ffafe7354d960daaa74de5bce"}
Jul 10 00:01:14.653964 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:14.653825    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b" event={"ID":"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb","Type":"ContainerStarted","Data":"7ff53241f667a3089dddddae7bd945fd8c2998e7d447eeb1ec097c5cf8431543"}
Jul 10 00:01:14.674135 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:14.674087    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b" podStartSLOduration=1.674071832 podStartE2EDuration="1.674071832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:01:14.672158663 +0000 UTC m=+1992.816089448" watchObservedRunningTime="2026-07-10 00:01:14.674071832 +0000 UTC m=+1992.818002617"
Jul 10 00:01:15.032455 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:15.032413    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:15.038104 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:15.038082    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:15.659224 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:15.659191    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:15.660089 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:15.660074    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:23.653819 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:23.653772    2580 patch_prober.go:28] interesting pod/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 10 00:01:23.654219 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:23.653867    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b" podUID="a1c36835-ac57-4118-bc68-cd681cd32f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 10 00:01:25.715333 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:25.715300    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1c36835-ac57-4118-bc68-cd681cd32f60" containerID="4cb61f6dddcc0354112b27ee10993fb0685d371c55c7848431918dfc503f1b5e" exitCode=0
Jul 10 00:01:25.715753 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:25.715375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b" event={"ID":"a1c36835-ac57-4118-bc68-cd681cd32f60","Type":"ContainerDied","Data":"4cb61f6dddcc0354112b27ee10993fb0685d371c55c7848431918dfc503f1b5e"}
Jul 10 00:01:25.715819 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:25.715802    2580 scope.go:117] "RemoveContainer" containerID="4cb61f6dddcc0354112b27ee10993fb0685d371c55c7848431918dfc503f1b5e"
Jul 10 00:01:25.716807 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:25.716757    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:25.716869 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:25.716848    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:25.716904 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:25.716879    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:26.722788 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:26.722754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b" event={"ID":"a1c36835-ac57-4118-bc68-cd681cd32f60","Type":"ContainerStarted","Data":"93119fc8708a9b123e66228d17db03bdbf5bbf9a2e75e9e08a23c897790769b2"}
Jul 10 00:01:27.639727 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:27.639689    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:27.640150 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:27.640123    2580 patch_prober.go:28] interesting pod/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 10 00:01:27.640359 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:27.640332    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b" podUID="a1c36835-ac57-4118-bc68-cd681cd32f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 10 00:01:28.641071 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:28.640550    2580 patch_prober.go:28] interesting pod/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 10 00:01:28.641071 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:28.640624    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b" podUID="a1c36835-ac57-4118-bc68-cd681cd32f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 10 00:01:29.640397 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:29.640360    2580 patch_prober.go:28] interesting pod/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 10 00:01:29.640668 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:29.640624    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b" podUID="a1c36835-ac57-4118-bc68-cd681cd32f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 10 00:01:30.297490 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:30.297466    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq"
Jul 10 00:01:30.695505 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:30.695382    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:30.695808 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:30.695783    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:30.696757 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:30.696734    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:30.778272 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:30.778237    2580 generic.go:356] "Generic (PLEG): container finished" podID="738aabed-ade7-4d4b-bb43-9e4a2454786a" containerID="225b412786bf671737a39a513acf52f62cc22c22852e99659487c5462d095b68" exitCode=137
Jul 10 00:01:30.778901 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:30.778880    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq"
Jul 10 00:01:30.779237 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:30.779184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq" event={"ID":"738aabed-ade7-4d4b-bb43-9e4a2454786a","Type":"ContainerDied","Data":"225b412786bf671737a39a513acf52f62cc22c22852e99659487c5462d095b68"}
Jul 10 00:01:30.779352 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:30.779320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq" event={"ID":"738aabed-ade7-4d4b-bb43-9e4a2454786a","Type":"ContainerDied","Data":"d36fd09ca9af75f544288d995c26083c63c69a409536822155a68b0ba6165b46"}
Jul 10 00:01:30.779419 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:30.779352    2580 scope.go:117] "RemoveContainer" containerID="225b412786bf671737a39a513acf52f62cc22c22852e99659487c5462d095b68"
Jul 10 00:01:30.788737 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:30.788720    2580 scope.go:117] "RemoveContainer" containerID="225b412786bf671737a39a513acf52f62cc22c22852e99659487c5462d095b68"
Jul 10 00:01:30.788995 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:01:30.788970    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225b412786bf671737a39a513acf52f62cc22c22852e99659487c5462d095b68\": container with ID starting with 225b412786bf671737a39a513acf52f62cc22c22852e99659487c5462d095b68 not found: ID does not exist" containerID="225b412786bf671737a39a513acf52f62cc22c22852e99659487c5462d095b68"
Jul 10 00:01:30.789092 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:30.789001    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225b412786bf671737a39a513acf52f62cc22c22852e99659487c5462d095b68"} err="failed to get container status \"225b412786bf671737a39a513acf52f62cc22c22852e99659487c5462d095b68\": rpc error: code = NotFound desc = could not find container \"225b412786bf671737a39a513acf52f62cc22c22852e99659487c5462d095b68\": container with ID starting with 225b412786bf671737a39a513acf52f62cc22c22852e99659487c5462d095b68 not found: ID does not exist"
Jul 10 00:01:30.796734 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:30.796704    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq"]
Jul 10 00:01:30.800064 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:30.800040    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727360-sgmpq"]
Jul 10 00:01:32.420467 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:32.420424    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="738aabed-ade7-4d4b-bb43-9e4a2454786a" path="/var/lib/kubelet/pods/738aabed-ade7-4d4b-bb43-9e4a2454786a/volumes"
Jul 10 00:01:34.802475 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:34.802442    2580 generic.go:356] "Generic (PLEG): container finished" podID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" containerID="4836af8814f0e44dff72907c061d7bea5a49a22ffafe7354d960daaa74de5bce" exitCode=0
Jul 10 00:01:34.802957 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:34.802515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b" event={"ID":"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb","Type":"ContainerDied","Data":"4836af8814f0e44dff72907c061d7bea5a49a22ffafe7354d960daaa74de5bce"}
Jul 10 00:01:34.802957 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:34.802940    2580 scope.go:117] "RemoveContainer" containerID="4836af8814f0e44dff72907c061d7bea5a49a22ffafe7354d960daaa74de5bce"
Jul 10 00:01:34.803955 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:34.803926    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:34.804163 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:34.803998    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:34.804163 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:34.804047    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:35.561226 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:35.561191    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"]
Jul 10 00:01:35.561684 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:35.561635    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b" podUID="a1c36835-ac57-4118-bc68-cd681cd32f60" containerName="istio-proxy" containerID="cri-o://93119fc8708a9b123e66228d17db03bdbf5bbf9a2e75e9e08a23c897790769b2" gracePeriod=30
Jul 10 00:01:35.809893 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:35.809857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b" event={"ID":"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb","Type":"ContainerStarted","Data":"dd0fbb92de79019d29189a64a25fd277a4df906fec8154d7e28152b8b534c2fd"}
Jul 10 00:01:36.032826 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:36.032782    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:36.033118 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:36.033092    2580 patch_prober.go:28] interesting pod/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Jul 10 00:01:36.033190 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:36.033145    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b" podUID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Jul 10 00:01:37.032890 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:37.032853    2580 patch_prober.go:28] interesting pod/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Jul 10 00:01:37.033573 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:37.033540    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b" podUID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Jul 10 00:01:38.032437 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:38.032389    2580 patch_prober.go:28] interesting pod/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Jul 10 00:01:38.032679 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:38.032457    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b" podUID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Jul 10 00:01:39.033115 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:39.033075    2580 patch_prober.go:28] interesting pod/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Jul 10 00:01:39.033719 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:39.033687    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b" podUID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Jul 10 00:01:40.036628 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.036595    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:40.037067 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.036919    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:40.037623 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.037602    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:40.817887 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.817863    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:40.842797 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.842709    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1c36835-ac57-4118-bc68-cd681cd32f60" containerID="93119fc8708a9b123e66228d17db03bdbf5bbf9a2e75e9e08a23c897790769b2" exitCode=0
Jul 10 00:01:40.842797 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.842755    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"
Jul 10 00:01:40.842797 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.842782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b" event={"ID":"a1c36835-ac57-4118-bc68-cd681cd32f60","Type":"ContainerDied","Data":"93119fc8708a9b123e66228d17db03bdbf5bbf9a2e75e9e08a23c897790769b2"}
Jul 10 00:01:40.843003 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.842842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b" event={"ID":"a1c36835-ac57-4118-bc68-cd681cd32f60","Type":"ContainerDied","Data":"6ee7b541d70e45f66c7157caab24ef31aa514ce72a5bba0b4dff4600e065ac3e"}
Jul 10 00:01:40.843003 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.842877    2580 scope.go:117] "RemoveContainer" containerID="93119fc8708a9b123e66228d17db03bdbf5bbf9a2e75e9e08a23c897790769b2"
Jul 10 00:01:40.853035 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.853011    2580 scope.go:117] "RemoveContainer" containerID="4cb61f6dddcc0354112b27ee10993fb0685d371c55c7848431918dfc503f1b5e"
Jul 10 00:01:40.863751 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.863731    2580 scope.go:117] "RemoveContainer" containerID="93119fc8708a9b123e66228d17db03bdbf5bbf9a2e75e9e08a23c897790769b2"
Jul 10 00:01:40.864080 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:01:40.864058    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93119fc8708a9b123e66228d17db03bdbf5bbf9a2e75e9e08a23c897790769b2\": container with ID starting with 93119fc8708a9b123e66228d17db03bdbf5bbf9a2e75e9e08a23c897790769b2 not found: ID does not exist" containerID="93119fc8708a9b123e66228d17db03bdbf5bbf9a2e75e9e08a23c897790769b2"
Jul 10 00:01:40.864152 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.864091    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93119fc8708a9b123e66228d17db03bdbf5bbf9a2e75e9e08a23c897790769b2"} err="failed to get container status \"93119fc8708a9b123e66228d17db03bdbf5bbf9a2e75e9e08a23c897790769b2\": rpc error: code = NotFound desc = could not find container \"93119fc8708a9b123e66228d17db03bdbf5bbf9a2e75e9e08a23c897790769b2\": container with ID starting with 93119fc8708a9b123e66228d17db03bdbf5bbf9a2e75e9e08a23c897790769b2 not found: ID does not exist"
Jul 10 00:01:40.864152 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.864111    2580 scope.go:117] "RemoveContainer" containerID="4cb61f6dddcc0354112b27ee10993fb0685d371c55c7848431918dfc503f1b5e"
Jul 10 00:01:40.864395 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:01:40.864371    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cb61f6dddcc0354112b27ee10993fb0685d371c55c7848431918dfc503f1b5e\": container with ID starting with 4cb61f6dddcc0354112b27ee10993fb0685d371c55c7848431918dfc503f1b5e not found: ID does not exist" containerID="4cb61f6dddcc0354112b27ee10993fb0685d371c55c7848431918dfc503f1b5e"
Jul 10 00:01:40.864451 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.864402    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cb61f6dddcc0354112b27ee10993fb0685d371c55c7848431918dfc503f1b5e"} err="failed to get container status \"4cb61f6dddcc0354112b27ee10993fb0685d371c55c7848431918dfc503f1b5e\": rpc error: code = NotFound desc = could not find container \"4cb61f6dddcc0354112b27ee10993fb0685d371c55c7848431918dfc503f1b5e\": container with ID starting with 4cb61f6dddcc0354112b27ee10993fb0685d371c55c7848431918dfc503f1b5e not found: ID does not exist"
Jul 10 00:01:40.911794 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.911753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-workload-certs\") pod \"a1c36835-ac57-4118-bc68-cd681cd32f60\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") "
Jul 10 00:01:40.911992 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.911839    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c36835-ac57-4118-bc68-cd681cd32f60-istiod-ca-cert\") pod \"a1c36835-ac57-4118-bc68-cd681cd32f60\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") "
Jul 10 00:01:40.911992 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.911888    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-envoy\") pod \"a1c36835-ac57-4118-bc68-cd681cd32f60\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") "
Jul 10 00:01:40.912113 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.911996    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-workload-socket\") pod \"a1c36835-ac57-4118-bc68-cd681cd32f60\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") "
Jul 10 00:01:40.912113 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.912054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-token\") pod \"a1c36835-ac57-4118-bc68-cd681cd32f60\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") "
Jul 10 00:01:40.912113 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.912085    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz55z\" (UniqueName: \"kubernetes.io/projected/a1c36835-ac57-4118-bc68-cd681cd32f60-kube-api-access-fz55z\") pod \"a1c36835-ac57-4118-bc68-cd681cd32f60\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") "
Jul 10 00:01:40.912113 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.912114    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-data\") pod \"a1c36835-ac57-4118-bc68-cd681cd32f60\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") "
Jul 10 00:01:40.912331 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.912164    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-credential-socket\") pod \"a1c36835-ac57-4118-bc68-cd681cd32f60\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") "
Jul 10 00:01:40.912331 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.912196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-podinfo\") pod \"a1c36835-ac57-4118-bc68-cd681cd32f60\" (UID: \"a1c36835-ac57-4118-bc68-cd681cd32f60\") "
Jul 10 00:01:40.912331 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.912110    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1c36835-ac57-4118-bc68-cd681cd32f60" (UID: "a1c36835-ac57-4118-bc68-cd681cd32f60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:40.912331 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.912297    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1c36835-ac57-4118-bc68-cd681cd32f60" (UID: "a1c36835-ac57-4118-bc68-cd681cd32f60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:40.912331 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.912323    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c36835-ac57-4118-bc68-cd681cd32f60-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1c36835-ac57-4118-bc68-cd681cd32f60" (UID: "a1c36835-ac57-4118-bc68-cd681cd32f60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 00:01:40.912603 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.912537    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-workload-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:40.912603 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.912559    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-workload-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:40.912603 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.912571    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c36835-ac57-4118-bc68-cd681cd32f60-istiod-ca-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:40.912603 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.912577    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1c36835-ac57-4118-bc68-cd681cd32f60" (UID: "a1c36835-ac57-4118-bc68-cd681cd32f60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:40.913314 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.913287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1c36835-ac57-4118-bc68-cd681cd32f60" (UID: "a1c36835-ac57-4118-bc68-cd681cd32f60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:40.914312 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.914287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1c36835-ac57-4118-bc68-cd681cd32f60" (UID: "a1c36835-ac57-4118-bc68-cd681cd32f60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:40.914312 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.914306    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1c36835-ac57-4118-bc68-cd681cd32f60" (UID: "a1c36835-ac57-4118-bc68-cd681cd32f60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:01:40.914758 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.914738    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1c36835-ac57-4118-bc68-cd681cd32f60" (UID: "a1c36835-ac57-4118-bc68-cd681cd32f60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 10 00:01:40.914989 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:40.914963    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c36835-ac57-4118-bc68-cd681cd32f60-kube-api-access-fz55z" (OuterVolumeSpecName: "kube-api-access-fz55z") pod "a1c36835-ac57-4118-bc68-cd681cd32f60" (UID: "a1c36835-ac57-4118-bc68-cd681cd32f60"). InnerVolumeSpecName "kube-api-access-fz55z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:01:41.013591 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:41.013550    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-token\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:41.013591 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:41.013585    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz55z\" (UniqueName: \"kubernetes.io/projected/a1c36835-ac57-4118-bc68-cd681cd32f60-kube-api-access-fz55z\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:41.013591 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:41.013598    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-data\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:41.013873 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:41.013610    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-credential-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:41.013873 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:41.013621    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-podinfo\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:41.013873 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:41.013632    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c36835-ac57-4118-bc68-cd681cd32f60-istio-envoy\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:41.167239 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:41.167208    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"]
Jul 10 00:01:41.170932 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:41.170905    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-04eddb-openshift-default-7f97cc57b5-5jx2b"]
Jul 10 00:01:42.286296 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:42.286262    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"]
Jul 10 00:01:42.419410 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:42.419375    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c36835-ac57-4118-bc68-cd681cd32f60" path="/var/lib/kubelet/pods/a1c36835-ac57-4118-bc68-cd681cd32f60/volumes"
Jul 10 00:01:42.857878 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:42.857808    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b" podUID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" containerName="istio-proxy" containerID="cri-o://dd0fbb92de79019d29189a64a25fd277a4df906fec8154d7e28152b8b534c2fd" gracePeriod=30
Jul 10 00:01:43.127632 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.127557    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"]
Jul 10 00:01:43.128160 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.128133    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="738aabed-ade7-4d4b-bb43-9e4a2454786a" containerName="cleanup"
Jul 10 00:01:43.128357 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.128318    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="738aabed-ade7-4d4b-bb43-9e4a2454786a" containerName="cleanup"
Jul 10 00:01:43.128476 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.128457    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c36835-ac57-4118-bc68-cd681cd32f60" containerName="istio-proxy"
Jul 10 00:01:43.128476 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.128476    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c36835-ac57-4118-bc68-cd681cd32f60" containerName="istio-proxy"
Jul 10 00:01:43.128632 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.128620    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c36835-ac57-4118-bc68-cd681cd32f60" containerName="istio-proxy"
Jul 10 00:01:43.128705 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.128631    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c36835-ac57-4118-bc68-cd681cd32f60" containerName="istio-proxy"
Jul 10 00:01:43.128705 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.128643    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="738aabed-ade7-4d4b-bb43-9e4a2454786a" containerName="cleanup"
Jul 10 00:01:43.132184 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.132158    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.134681 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.134639    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-4c394e-openshift-default-dockercfg-26xzh\""
Jul 10 00:01:43.143485 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.143458    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"]
Jul 10 00:01:43.233275 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.233237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-podinfo\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.233578 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.233553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76q9w\" (UniqueName: \"kubernetes.io/projected/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-kube-api-access-76q9w\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.233811 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.233771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-envoy\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.233986 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.233965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-credential-socket\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.234082 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.234004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istiod-ca-cert\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.234082 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.234033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-workload-socket\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.234082 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.234059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-workload-certs\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.234236 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.234100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-token\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.234236 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.234148    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-data\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.335469 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.335432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-podinfo\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.335469 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.335479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76q9w\" (UniqueName: \"kubernetes.io/projected/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-kube-api-access-76q9w\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.336072 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.335528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-envoy\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.336072 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.335561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-credential-socket\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.336072 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.335584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istiod-ca-cert\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.336072 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.335612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-workload-socket\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.336072 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.335643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-workload-certs\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.336072 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.335702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-token\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.336072 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.335750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-data\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.336462 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.336429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istiod-ca-cert\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.336679 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.336632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-workload-certs\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.336770 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.336723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-data\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.336950 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.336926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-workload-socket\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.337022 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.336968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-credential-socket\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.338190 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.338166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-envoy\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.338395 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.338376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-podinfo\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.343711 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.343685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-token\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.343852 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.343834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76q9w\" (UniqueName: \"kubernetes.io/projected/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-kube-api-access-76q9w\") pod \"e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") " pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.444290 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.444188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:43.581887 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.581860    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"]
Jul 10 00:01:43.583812 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:01:43.583789    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc569f4_ee41_41ee_a4ea_9d4e06cb6688.slice/crio-3ac33aebbce33efd8bd2cb42170f261b615345e38ce139bd7e98fd8c8d186e85 WatchSource:0}: Error finding container 3ac33aebbce33efd8bd2cb42170f261b615345e38ce139bd7e98fd8c8d186e85: Status 404 returned error can't find the container with id 3ac33aebbce33efd8bd2cb42170f261b615345e38ce139bd7e98fd8c8d186e85
Jul 10 00:01:43.586304 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.586270    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:43.586410 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.586355    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:43.586410 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.586400    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:43.866424 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.866379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" event={"ID":"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688","Type":"ContainerStarted","Data":"cd176b6ebdaede63e1ea0a12067f8407d9228673c5cde2ac3a361e41c25cf279"}
Jul 10 00:01:43.866604 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.866447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" event={"ID":"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688","Type":"ContainerStarted","Data":"3ac33aebbce33efd8bd2cb42170f261b615345e38ce139bd7e98fd8c8d186e85"}
Jul 10 00:01:43.885187 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:43.885136    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" podStartSLOduration=0.885119428 podStartE2EDuration="885.119428ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:01:43.883553655 +0000 UTC m=+2022.027484481" watchObservedRunningTime="2026-07-10 00:01:43.885119428 +0000 UTC m=+2022.029050214"
Jul 10 00:01:44.444666 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:44.444620    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:44.449860 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:44.449836    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:44.871424 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:44.871389    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:44.872395 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:44.872373    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:01:48.114617 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.114589    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:48.294806 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.294773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-workload-socket\") pod \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") "
Jul 10 00:01:48.294963 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.294822    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-podinfo\") pod \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") "
Jul 10 00:01:48.294963 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.294862    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-workload-certs\") pod \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") "
Jul 10 00:01:48.294963 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.294885    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-credential-socket\") pod \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") "
Jul 10 00:01:48.294963 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.294927    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-envoy\") pod \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") "
Jul 10 00:01:48.294963 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.294954    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh72s\" (UniqueName: \"kubernetes.io/projected/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-kube-api-access-qh72s\") pod \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") "
Jul 10 00:01:48.295320 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.295004    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istiod-ca-cert\") pod \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") "
Jul 10 00:01:48.295320 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.295047    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-data\") pod \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") "
Jul 10 00:01:48.295477 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.295458    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-token\") pod \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\" (UID: \"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb\") "
Jul 10 00:01:48.295529 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.295089    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" (UID: "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:48.295529 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.295213    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" (UID: "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:48.295634 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.295226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" (UID: "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:48.295634 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.295400    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" (UID: "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 00:01:48.295847 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.295825    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istiod-ca-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:48.295847 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.295848    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-workload-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:48.295984 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.295862    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-workload-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:48.295984 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.295876    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-credential-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:48.295984 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.295910    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-data" (OuterVolumeSpecName: "istio-data") pod "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" (UID: "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:48.297145 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.297120    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-kube-api-access-qh72s" (OuterVolumeSpecName: "kube-api-access-qh72s") pod "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" (UID: "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb"). InnerVolumeSpecName "kube-api-access-qh72s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:01:48.297547 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.297530    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" (UID: "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:01:48.297547 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.297536    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-token" (OuterVolumeSpecName: "istio-token") pod "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" (UID: "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:01:48.297677 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.297529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" (UID: "8fe74446-ebb5-4f10-8bd2-6bd7df25eceb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 10 00:01:48.396835 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.396793    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-podinfo\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:48.396835 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.396826    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-envoy\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:48.396835 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.396843    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh72s\" (UniqueName: \"kubernetes.io/projected/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-kube-api-access-qh72s\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:48.397074 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.396857    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-data\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:48.397074 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.396868    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb-istio-token\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:01:48.894933 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.894901    2580 generic.go:356] "Generic (PLEG): container finished" podID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" containerID="dd0fbb92de79019d29189a64a25fd277a4df906fec8154d7e28152b8b534c2fd" exitCode=0
Jul 10 00:01:48.895089 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.894952    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"
Jul 10 00:01:48.895089 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.894986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b" event={"ID":"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb","Type":"ContainerDied","Data":"dd0fbb92de79019d29189a64a25fd277a4df906fec8154d7e28152b8b534c2fd"}
Jul 10 00:01:48.895089 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.895044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b" event={"ID":"8fe74446-ebb5-4f10-8bd2-6bd7df25eceb","Type":"ContainerDied","Data":"7ff53241f667a3089dddddae7bd945fd8c2998e7d447eeb1ec097c5cf8431543"}
Jul 10 00:01:48.895089 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.895069    2580 scope.go:117] "RemoveContainer" containerID="dd0fbb92de79019d29189a64a25fd277a4df906fec8154d7e28152b8b534c2fd"
Jul 10 00:01:48.904224 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.904203    2580 scope.go:117] "RemoveContainer" containerID="4836af8814f0e44dff72907c061d7bea5a49a22ffafe7354d960daaa74de5bce"
Jul 10 00:01:48.911004 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.910981    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"]
Jul 10 00:01:48.914240 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.914217    2580 scope.go:117] "RemoveContainer" containerID="dd0fbb92de79019d29189a64a25fd277a4df906fec8154d7e28152b8b534c2fd"
Jul 10 00:01:48.914479 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.914461    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9ed59f-openshift-default-c4d4588b9-47h4b"]
Jul 10 00:01:48.914568 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:01:48.914519    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0fbb92de79019d29189a64a25fd277a4df906fec8154d7e28152b8b534c2fd\": container with ID starting with dd0fbb92de79019d29189a64a25fd277a4df906fec8154d7e28152b8b534c2fd not found: ID does not exist" containerID="dd0fbb92de79019d29189a64a25fd277a4df906fec8154d7e28152b8b534c2fd"
Jul 10 00:01:48.914568 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.914547    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0fbb92de79019d29189a64a25fd277a4df906fec8154d7e28152b8b534c2fd"} err="failed to get container status \"dd0fbb92de79019d29189a64a25fd277a4df906fec8154d7e28152b8b534c2fd\": rpc error: code = NotFound desc = could not find container \"dd0fbb92de79019d29189a64a25fd277a4df906fec8154d7e28152b8b534c2fd\": container with ID starting with dd0fbb92de79019d29189a64a25fd277a4df906fec8154d7e28152b8b534c2fd not found: ID does not exist"
Jul 10 00:01:48.914568 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.914566    2580 scope.go:117] "RemoveContainer" containerID="4836af8814f0e44dff72907c061d7bea5a49a22ffafe7354d960daaa74de5bce"
Jul 10 00:01:48.914871 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:01:48.914854    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4836af8814f0e44dff72907c061d7bea5a49a22ffafe7354d960daaa74de5bce\": container with ID starting with 4836af8814f0e44dff72907c061d7bea5a49a22ffafe7354d960daaa74de5bce not found: ID does not exist" containerID="4836af8814f0e44dff72907c061d7bea5a49a22ffafe7354d960daaa74de5bce"
Jul 10 00:01:48.914923 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:48.914881    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4836af8814f0e44dff72907c061d7bea5a49a22ffafe7354d960daaa74de5bce"} err="failed to get container status \"4836af8814f0e44dff72907c061d7bea5a49a22ffafe7354d960daaa74de5bce\": rpc error: code = NotFound desc = could not find container \"4836af8814f0e44dff72907c061d7bea5a49a22ffafe7354d960daaa74de5bce\": container with ID starting with 4836af8814f0e44dff72907c061d7bea5a49a22ffafe7354d960daaa74de5bce not found: ID does not exist"
Jul 10 00:01:50.419837 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.419805    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" path="/var/lib/kubelet/pods/8fe74446-ebb5-4f10-8bd2-6bd7df25eceb/volumes"
Jul 10 00:01:50.810843 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.810794    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"]
Jul 10 00:01:50.811430 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.811409    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c36835-ac57-4118-bc68-cd681cd32f60" containerName="istio-proxy"
Jul 10 00:01:50.811430 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.811430    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c36835-ac57-4118-bc68-cd681cd32f60" containerName="istio-proxy"
Jul 10 00:01:50.811599 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.811445    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" containerName="istio-proxy"
Jul 10 00:01:50.811599 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.811454    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" containerName="istio-proxy"
Jul 10 00:01:50.811599 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.811470    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" containerName="istio-proxy"
Jul 10 00:01:50.811599 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.811479    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" containerName="istio-proxy"
Jul 10 00:01:50.811599 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.811566    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" containerName="istio-proxy"
Jul 10 00:01:50.811599 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.811581    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe74446-ebb5-4f10-8bd2-6bd7df25eceb" containerName="istio-proxy"
Jul 10 00:01:50.815078 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.815053    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:50.817514 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.817496    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b0ecf1-openshift-default-dockercfg-kkvhh\""
Jul 10 00:01:50.824635 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.824613    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"]
Jul 10 00:01:50.920244 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.920175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm5q5\" (UniqueName: \"kubernetes.io/projected/dac4d79e-ebfe-453a-8a11-cd3082264301-kube-api-access-qm5q5\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:50.920536 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.920510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-workload-certs\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:50.920536 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.920550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-token\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:50.920536 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.920583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-credential-socket\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:50.920931 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.920606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-podinfo\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:50.920931 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.920711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-envoy\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:50.920931 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.920759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac4d79e-ebfe-453a-8a11-cd3082264301-istiod-ca-cert\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:50.920931 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.920843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-data\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:50.920931 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:50.920904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-workload-socket\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022036 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.021992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac4d79e-ebfe-453a-8a11-cd3082264301-istiod-ca-cert\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022237 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.022062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-data\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022237 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.022108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-workload-socket\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022237 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.022168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm5q5\" (UniqueName: \"kubernetes.io/projected/dac4d79e-ebfe-453a-8a11-cd3082264301-kube-api-access-qm5q5\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022237 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.022193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-workload-certs\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022237 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.022218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-token\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022532 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.022249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-credential-socket\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022532 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.022274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-podinfo\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022532 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.022317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-envoy\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022532 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.022470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-data\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022742 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.022671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-workload-socket\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022742 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.022726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac4d79e-ebfe-453a-8a11-cd3082264301-istiod-ca-cert\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022892 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.022872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-credential-socket\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.022965 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.022936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-workload-certs\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.024855 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.024831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-podinfo\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.024949 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.024879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-envoy\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.029810 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.029783    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm5q5\" (UniqueName: \"kubernetes.io/projected/dac4d79e-ebfe-453a-8a11-cd3082264301-kube-api-access-qm5q5\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.029923 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.029818    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-token\") pod \"e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") " pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.126731 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.126618    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:51.267456 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.267426    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"]
Jul 10 00:01:51.269161 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:01:51.269133    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac4d79e_ebfe_453a_8a11_cd3082264301.slice/crio-90c77397661413555826d27876877415b3a3d5857e63e68102a5448337590c4e WatchSource:0}: Error finding container 90c77397661413555826d27876877415b3a3d5857e63e68102a5448337590c4e: Status 404 returned error can't find the container with id 90c77397661413555826d27876877415b3a3d5857e63e68102a5448337590c4e
Jul 10 00:01:51.271280 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.271240    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:51.271370 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.271312    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:51.271370 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.271345    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:51.914557 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.914522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" event={"ID":"dac4d79e-ebfe-453a-8a11-cd3082264301","Type":"ContainerStarted","Data":"9a8a3bda5368d68beb3bf56b6f3c664e5d9dcf181e3f118708a4d0468d22e1a0"}
Jul 10 00:01:51.914986 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.914573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" event={"ID":"dac4d79e-ebfe-453a-8a11-cd3082264301","Type":"ContainerStarted","Data":"90c77397661413555826d27876877415b3a3d5857e63e68102a5448337590c4e"}
Jul 10 00:01:51.933495 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:51.933435    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" podStartSLOduration=1.933416701 podStartE2EDuration="1.933416701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:01:51.931663862 +0000 UTC m=+2030.075594641" watchObservedRunningTime="2026-07-10 00:01:51.933416701 +0000 UTC m=+2030.077347478"
Jul 10 00:01:52.127863 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:52.127823    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:52.133265 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:52.133236    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:52.919276 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:52.919234    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:52.920357 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:52.920338    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:01:59.872605 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:59.872516    2580 patch_prober.go:28] interesting pod/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 10 00:01:59.872605 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:59.872574    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 10 00:01:59.966124 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:59.966081    2580 generic.go:356] "Generic (PLEG): container finished" podID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerID="cd176b6ebdaede63e1ea0a12067f8407d9228673c5cde2ac3a361e41c25cf279" exitCode=0
Jul 10 00:01:59.966339 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:59.966247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" event={"ID":"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688","Type":"ContainerDied","Data":"cd176b6ebdaede63e1ea0a12067f8407d9228673c5cde2ac3a361e41c25cf279"}
Jul 10 00:01:59.966846 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:59.966823    2580 scope.go:117] "RemoveContainer" containerID="cd176b6ebdaede63e1ea0a12067f8407d9228673c5cde2ac3a361e41c25cf279"
Jul 10 00:01:59.969192 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:59.969145    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:59.969451 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:59.969419    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:01:59.969564 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:01:59.969545    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:02:00.974499 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:00.974465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" event={"ID":"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688","Type":"ContainerStarted","Data":"50e5a8e1d6f2ea520db1ca24db28674c63174e3ce373ee84023d0dae6b8d7084"}
Jul 10 00:02:01.445558 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:01.445106    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:02:01.445558 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:01.445244    2580 patch_prober.go:28] interesting pod/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 10 00:02:01.445558 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:01.445291    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 10 00:02:02.445723 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:02.445677    2580 patch_prober.go:28] interesting pod/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 10 00:02:02.446229 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:02.445747    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 10 00:02:03.445592 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:03.445546    2580 patch_prober.go:28] interesting pod/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 10 00:02:03.445834 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:03.445613    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 10 00:02:04.444992 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:04.444954    2580 patch_prober.go:28] interesting pod/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 10 00:02:04.445386 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:04.445355    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 10 00:02:05.445698 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:05.445234    2580 patch_prober.go:28] interesting pod/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 10 00:02:05.445698 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:05.445299    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 10 00:02:06.448942 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:06.448914    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:02:06.449327 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:06.449287    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:02:06.450104 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:06.450084    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:02:07.920737 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:07.920696    2580 patch_prober.go:28] interesting pod/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Jul 10 00:02:07.921151 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:07.920770    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Jul 10 00:02:08.013541 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:08.013506    2580 generic.go:356] "Generic (PLEG): container finished" podID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerID="9a8a3bda5368d68beb3bf56b6f3c664e5d9dcf181e3f118708a4d0468d22e1a0" exitCode=0
Jul 10 00:02:08.013767 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:08.013580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" event={"ID":"dac4d79e-ebfe-453a-8a11-cd3082264301","Type":"ContainerDied","Data":"9a8a3bda5368d68beb3bf56b6f3c664e5d9dcf181e3f118708a4d0468d22e1a0"}
Jul 10 00:02:08.014161 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:08.014136    2580 scope.go:117] "RemoveContainer" containerID="9a8a3bda5368d68beb3bf56b6f3c664e5d9dcf181e3f118708a4d0468d22e1a0"
Jul 10 00:02:08.015149 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:08.015124    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:02:08.015243 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:08.015202    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:02:08.015293 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:08.015259    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 10 00:02:09.021396 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:09.021361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" event={"ID":"dac4d79e-ebfe-453a-8a11-cd3082264301","Type":"ContainerStarted","Data":"1b14cce6276cf092657f9b26dd1ae6b6e031560c54f1ea4ad2dec73737f45db7"}
Jul 10 00:02:09.127670 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:09.127613    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:02:09.127861 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:09.127829    2580 patch_prober.go:28] interesting pod/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Jul 10 00:02:09.127938 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:09.127886    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Jul 10 00:02:10.127751 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:10.127700    2580 patch_prober.go:28] interesting pod/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Jul 10 00:02:10.128264 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:10.127784    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Jul 10 00:02:11.128102 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:11.128054    2580 patch_prober.go:28] interesting pod/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Jul 10 00:02:11.128595 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:11.128141    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Jul 10 00:02:12.127105 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:12.127070    2580 patch_prober.go:28] interesting pod/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Jul 10 00:02:12.127300 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:12.127139    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Jul 10 00:02:13.127109 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:13.127070    2580 patch_prober.go:28] interesting pod/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Jul 10 00:02:13.127599 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:13.127137    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Jul 10 00:02:14.131706 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:14.131672    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:02:14.132127 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:14.132019    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:02:14.132773 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:14.132750    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:02:27.586789 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.586748    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"]
Jul 10 00:02:27.589988 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.589970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.593034 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.593012    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4c394e\"/\"kube-root-ca.crt\""
Jul 10 00:02:27.593034 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.593025    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4c394e\"/\"openshift-service-ca.crt\""
Jul 10 00:02:27.594250 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.594232    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4c394e\"/\"default-dockercfg-98dhz\""
Jul 10 00:02:27.594321 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.594257    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4c394e\"/\"auth-test-model-4c394e-kserve-self-signed-certs\""
Jul 10 00:02:27.603068 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.603048    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"]
Jul 10 00:02:27.657967 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.657923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-model-cache\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.658129 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.657992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-tmp-dir\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.658129 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.658063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-dshm\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.658129 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.658089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93319e02-92e2-45c2-9e1a-3a1087646030-tls-certs\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.658129 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.658115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-home\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.658292 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.658164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwhpv\" (UniqueName: \"kubernetes.io/projected/93319e02-92e2-45c2-9e1a-3a1087646030-kube-api-access-zwhpv\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.759149 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.759115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwhpv\" (UniqueName: \"kubernetes.io/projected/93319e02-92e2-45c2-9e1a-3a1087646030-kube-api-access-zwhpv\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.759354 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.759159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-model-cache\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.759354 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.759300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-tmp-dir\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.759492 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.759418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-dshm\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.759492 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.759446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93319e02-92e2-45c2-9e1a-3a1087646030-tls-certs\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.759492 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.759478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-home\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.759618 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.759481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-model-cache\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.759710 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.759690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-tmp-dir\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.759752 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.759696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-home\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.761708 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.761678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-dshm\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.761914 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.761896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93319e02-92e2-45c2-9e1a-3a1087646030-tls-certs\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.766884 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.766866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwhpv\" (UniqueName: \"kubernetes.io/projected/93319e02-92e2-45c2-9e1a-3a1087646030-kube-api-access-zwhpv\") pod \"auth-test-model-4c394e-kserve-54557f699f-85xml\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") " pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:27.899233 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:27.899144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:28.038477 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:28.038447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"]
Jul 10 00:02:28.040253 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:02:28.040225    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93319e02_92e2_45c2_9e1a_3a1087646030.slice/crio-80a67956b034bb448a4613a45fcf0e98479b3deff4c0f0de1238d159f23f15b9 WatchSource:0}: Error finding container 80a67956b034bb448a4613a45fcf0e98479b3deff4c0f0de1238d159f23f15b9: Status 404 returned error can't find the container with id 80a67956b034bb448a4613a45fcf0e98479b3deff4c0f0de1238d159f23f15b9
Jul 10 00:02:28.144472 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:28.144440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml" event={"ID":"93319e02-92e2-45c2-9e1a-3a1087646030","Type":"ContainerStarted","Data":"91efdb8d441864e6d887fa05514ffb94e1c7f6cf31e1788511709894eee4338e"}
Jul 10 00:02:28.144642 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:28.144481    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml" event={"ID":"93319e02-92e2-45c2-9e1a-3a1087646030","Type":"ContainerStarted","Data":"80a67956b034bb448a4613a45fcf0e98479b3deff4c0f0de1238d159f23f15b9"}
Jul 10 00:02:28.161003 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:28.160906    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml" podStartSLOduration=1.160887513 podStartE2EDuration="1.160887513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:02:28.158455278 +0000 UTC m=+2066.302386078" watchObservedRunningTime="2026-07-10 00:02:28.160887513 +0000 UTC m=+2066.304818329"
Jul 10 00:02:37.899906 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:37.899870    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:37.900365 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:37.899919    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:37.912218 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:37.912189    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:38.207057 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:38.206971    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:02:42.882179 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:42.882141    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"]
Jul 10 00:02:42.884623 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:42.884605    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:42.888490 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:42.888263    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b0ecf1\"/\"openshift-service-ca.crt\""
Jul 10 00:02:42.888490 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:42.888296    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b0ecf1\"/\"default-dockercfg-5ssxx\""
Jul 10 00:02:42.888490 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:42.888333    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b0ecf1\"/\"auth-test-model-b0ecf1-kserve-self-signed-certs\""
Jul 10 00:02:42.888490 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:42.888264    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b0ecf1\"/\"kube-root-ca.crt\""
Jul 10 00:02:42.893689 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:42.893640    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"]
Jul 10 00:02:42.997753 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:42.997722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-home\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:42.997933 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:42.997764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-model-cache\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:42.997933 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:42.997800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m972p\" (UniqueName: \"kubernetes.io/projected/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-kube-api-access-m972p\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:42.997933 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:42.997858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-dshm\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:42.997933 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:42.997897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-tls-certs\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:42.997933 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:42.997923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-tmp-dir\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.098592 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.098558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-home\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.098592 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.098605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-model-cache\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.098867 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.098636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m972p\" (UniqueName: \"kubernetes.io/projected/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-kube-api-access-m972p\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.098867 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.098722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-dshm\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.098867 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.098745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-tls-certs\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.098867 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.098770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-tmp-dir\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.099085 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.099060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-home\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.099146 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.099098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-model-cache\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.099199 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.099148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-tmp-dir\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.101077 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.101053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-dshm\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.101301 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.101282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-tls-certs\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.106339 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.106319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m972p\" (UniqueName: \"kubernetes.io/projected/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-kube-api-access-m972p\") pod \"auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") " pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.196044 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.195956    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:43.332689 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:43.332636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"]
Jul 10 00:02:43.334279 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:02:43.334249    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4c54c13_a7e4_4f59_a3a2_9ecc9841c0f2.slice/crio-2e8c7ace7563313c1a94fb038d5855bb0feb511aa58db21174f46f4d6c13ccaa WatchSource:0}: Error finding container 2e8c7ace7563313c1a94fb038d5855bb0feb511aa58db21174f46f4d6c13ccaa: Status 404 returned error can't find the container with id 2e8c7ace7563313c1a94fb038d5855bb0feb511aa58db21174f46f4d6c13ccaa
Jul 10 00:02:44.232288 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:44.232241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d" event={"ID":"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2","Type":"ContainerStarted","Data":"e85a4de256fef8594834ccb3f6bc91274b26a362cca1d116e7a9ad3e63ec3c03"}
Jul 10 00:02:44.232733 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:44.232300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d" event={"ID":"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2","Type":"ContainerStarted","Data":"2e8c7ace7563313c1a94fb038d5855bb0feb511aa58db21174f46f4d6c13ccaa"}
Jul 10 00:02:44.251371 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:44.251315    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d" podStartSLOduration=2.251299153 podStartE2EDuration="2.251299153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:02:44.249326919 +0000 UTC m=+2082.393257705" watchObservedRunningTime="2026-07-10 00:02:44.251299153 +0000 UTC m=+2082.395229975"
Jul 10 00:02:53.197066 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:53.197020    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:53.197450 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:53.197109    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:53.209557 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:53.209532    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:02:53.292885 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:02:53.292851    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:04:30.854411 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:30.854368    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"]
Jul 10 00:04:31.794489 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:31.794456    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"]
Jul 10 00:04:45.875004 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:45.874938    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml" podUID="93319e02-92e2-45c2-9e1a-3a1087646030" containerName="main" containerID="cri-o://91efdb8d441864e6d887fa05514ffb94e1c7f6cf31e1788511709894eee4338e" gracePeriod=45
Jul 10 00:04:46.126158 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.126092    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:04:46.198906 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.198876    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93319e02-92e2-45c2-9e1a-3a1087646030-tls-certs\") pod \"93319e02-92e2-45c2-9e1a-3a1087646030\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") "
Jul 10 00:04:46.199111 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.198924    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-home\") pod \"93319e02-92e2-45c2-9e1a-3a1087646030\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") "
Jul 10 00:04:46.199111 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.198951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-model-cache\") pod \"93319e02-92e2-45c2-9e1a-3a1087646030\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") "
Jul 10 00:04:46.199111 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.198975    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-tmp-dir\") pod \"93319e02-92e2-45c2-9e1a-3a1087646030\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") "
Jul 10 00:04:46.199111 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.199050    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwhpv\" (UniqueName: \"kubernetes.io/projected/93319e02-92e2-45c2-9e1a-3a1087646030-kube-api-access-zwhpv\") pod \"93319e02-92e2-45c2-9e1a-3a1087646030\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") "
Jul 10 00:04:46.199347 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.199115    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-dshm\") pod \"93319e02-92e2-45c2-9e1a-3a1087646030\" (UID: \"93319e02-92e2-45c2-9e1a-3a1087646030\") "
Jul 10 00:04:46.199347 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.199238    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93319e02-92e2-45c2-9e1a-3a1087646030" (UID: "93319e02-92e2-45c2-9e1a-3a1087646030"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:04:46.199347 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.199224    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-home" (OuterVolumeSpecName: "home") pod "93319e02-92e2-45c2-9e1a-3a1087646030" (UID: "93319e02-92e2-45c2-9e1a-3a1087646030"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:04:46.199347 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.199268    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-model-cache" (OuterVolumeSpecName: "model-cache") pod "93319e02-92e2-45c2-9e1a-3a1087646030" (UID: "93319e02-92e2-45c2-9e1a-3a1087646030"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:04:46.199547 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.199490    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-home\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:04:46.199547 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.199507    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-model-cache\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:04:46.199547 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.199524    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-tmp-dir\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:04:46.201303 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.201276    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-dshm" (OuterVolumeSpecName: "dshm") pod "93319e02-92e2-45c2-9e1a-3a1087646030" (UID: "93319e02-92e2-45c2-9e1a-3a1087646030"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:04:46.201303 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.201296    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93319e02-92e2-45c2-9e1a-3a1087646030-kube-api-access-zwhpv" (OuterVolumeSpecName: "kube-api-access-zwhpv") pod "93319e02-92e2-45c2-9e1a-3a1087646030" (UID: "93319e02-92e2-45c2-9e1a-3a1087646030"). InnerVolumeSpecName "kube-api-access-zwhpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:04:46.201463 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.201323    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93319e02-92e2-45c2-9e1a-3a1087646030-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93319e02-92e2-45c2-9e1a-3a1087646030" (UID: "93319e02-92e2-45c2-9e1a-3a1087646030"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 00:04:46.300984 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.300946    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwhpv\" (UniqueName: \"kubernetes.io/projected/93319e02-92e2-45c2-9e1a-3a1087646030-kube-api-access-zwhpv\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:04:46.300984 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.300976    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93319e02-92e2-45c2-9e1a-3a1087646030-dshm\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:04:46.300984 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.300986    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93319e02-92e2-45c2-9e1a-3a1087646030-tls-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:04:46.815295 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.815214    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d" podUID="c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2" containerName="main" containerID="cri-o://e85a4de256fef8594834ccb3f6bc91274b26a362cca1d116e7a9ad3e63ec3c03" gracePeriod=45
Jul 10 00:04:46.875711 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.875668    2580 generic.go:356] "Generic (PLEG): container finished" podID="93319e02-92e2-45c2-9e1a-3a1087646030" containerID="91efdb8d441864e6d887fa05514ffb94e1c7f6cf31e1788511709894eee4338e" exitCode=0
Jul 10 00:04:46.875711 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.875711    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml" event={"ID":"93319e02-92e2-45c2-9e1a-3a1087646030","Type":"ContainerDied","Data":"91efdb8d441864e6d887fa05514ffb94e1c7f6cf31e1788511709894eee4338e"}
Jul 10 00:04:46.876304 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.875740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml" event={"ID":"93319e02-92e2-45c2-9e1a-3a1087646030","Type":"ContainerDied","Data":"80a67956b034bb448a4613a45fcf0e98479b3deff4c0f0de1238d159f23f15b9"}
Jul 10 00:04:46.876304 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.875741    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"
Jul 10 00:04:46.876304 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.875756    2580 scope.go:117] "RemoveContainer" containerID="91efdb8d441864e6d887fa05514ffb94e1c7f6cf31e1788511709894eee4338e"
Jul 10 00:04:46.891066 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.891040    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"]
Jul 10 00:04:46.894447 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.894425    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/auth-test-model-4c394e-kserve-54557f699f-85xml"]
Jul 10 00:04:46.927044 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.927017    2580 scope.go:117] "RemoveContainer" containerID="91efdb8d441864e6d887fa05514ffb94e1c7f6cf31e1788511709894eee4338e"
Jul 10 00:04:46.927418 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:04:46.927397    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91efdb8d441864e6d887fa05514ffb94e1c7f6cf31e1788511709894eee4338e\": container with ID starting with 91efdb8d441864e6d887fa05514ffb94e1c7f6cf31e1788511709894eee4338e not found: ID does not exist" containerID="91efdb8d441864e6d887fa05514ffb94e1c7f6cf31e1788511709894eee4338e"
Jul 10 00:04:46.927483 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:46.927431    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91efdb8d441864e6d887fa05514ffb94e1c7f6cf31e1788511709894eee4338e"} err="failed to get container status \"91efdb8d441864e6d887fa05514ffb94e1c7f6cf31e1788511709894eee4338e\": rpc error: code = NotFound desc = could not find container \"91efdb8d441864e6d887fa05514ffb94e1c7f6cf31e1788511709894eee4338e\": container with ID starting with 91efdb8d441864e6d887fa05514ffb94e1c7f6cf31e1788511709894eee4338e not found: ID does not exist"
Jul 10 00:04:47.068996 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.068923    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:04:47.109124 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.109091    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-model-cache\") pod \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") "
Jul 10 00:04:47.109318 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.109141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-home\") pod \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") "
Jul 10 00:04:47.109318 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.109192    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-tls-certs\") pod \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") "
Jul 10 00:04:47.109318 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.109227    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m972p\" (UniqueName: \"kubernetes.io/projected/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-kube-api-access-m972p\") pod \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") "
Jul 10 00:04:47.109318 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.109286    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-dshm\") pod \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") "
Jul 10 00:04:47.109318 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.109314    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-tmp-dir\") pod \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\" (UID: \"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2\") "
Jul 10 00:04:47.109582 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.109390    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2" (UID: "c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:04:47.109582 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.109455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-home" (OuterVolumeSpecName: "home") pod "c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2" (UID: "c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:04:47.109756 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.109643    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2" (UID: "c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:04:47.109756 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.109694    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-model-cache\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:04:47.109756 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.109720    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-home\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:04:47.111479 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.111451    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2" (UID: "c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 00:04:47.111818 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.111798    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-dshm" (OuterVolumeSpecName: "dshm") pod "c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2" (UID: "c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:04:47.111873 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.111835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-kube-api-access-m972p" (OuterVolumeSpecName: "kube-api-access-m972p") pod "c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2" (UID: "c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2"). InnerVolumeSpecName "kube-api-access-m972p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:04:47.210422 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.210385    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-tls-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:04:47.210422 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.210413    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m972p\" (UniqueName: \"kubernetes.io/projected/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-kube-api-access-m972p\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:04:47.210422 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.210424    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-dshm\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:04:47.210790 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.210453    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2-tmp-dir\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:04:47.883827 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.883790    2580 generic.go:356] "Generic (PLEG): container finished" podID="c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2" containerID="e85a4de256fef8594834ccb3f6bc91274b26a362cca1d116e7a9ad3e63ec3c03" exitCode=0
Jul 10 00:04:47.884239 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.883839    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"
Jul 10 00:04:47.884239 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.883853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d" event={"ID":"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2","Type":"ContainerDied","Data":"e85a4de256fef8594834ccb3f6bc91274b26a362cca1d116e7a9ad3e63ec3c03"}
Jul 10 00:04:47.884239 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.883884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d" event={"ID":"c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2","Type":"ContainerDied","Data":"2e8c7ace7563313c1a94fb038d5855bb0feb511aa58db21174f46f4d6c13ccaa"}
Jul 10 00:04:47.884239 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.883904    2580 scope.go:117] "RemoveContainer" containerID="e85a4de256fef8594834ccb3f6bc91274b26a362cca1d116e7a9ad3e63ec3c03"
Jul 10 00:04:47.893621 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.893604    2580 scope.go:117] "RemoveContainer" containerID="e85a4de256fef8594834ccb3f6bc91274b26a362cca1d116e7a9ad3e63ec3c03"
Jul 10 00:04:47.893912 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:04:47.893882    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85a4de256fef8594834ccb3f6bc91274b26a362cca1d116e7a9ad3e63ec3c03\": container with ID starting with e85a4de256fef8594834ccb3f6bc91274b26a362cca1d116e7a9ad3e63ec3c03 not found: ID does not exist" containerID="e85a4de256fef8594834ccb3f6bc91274b26a362cca1d116e7a9ad3e63ec3c03"
Jul 10 00:04:47.894009 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.893909    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85a4de256fef8594834ccb3f6bc91274b26a362cca1d116e7a9ad3e63ec3c03"} err="failed to get container status \"e85a4de256fef8594834ccb3f6bc91274b26a362cca1d116e7a9ad3e63ec3c03\": rpc error: code = NotFound desc = could not find container \"e85a4de256fef8594834ccb3f6bc91274b26a362cca1d116e7a9ad3e63ec3c03\": container with ID starting with e85a4de256fef8594834ccb3f6bc91274b26a362cca1d116e7a9ad3e63ec3c03 not found: ID does not exist"
Jul 10 00:04:47.904807 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.904781    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"]
Jul 10 00:04:47.908420 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:47.908398    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/auth-test-model-b0ecf1-kserve-57f89fc87f-5hc4d"]
Jul 10 00:04:48.419583 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:48.419552    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93319e02-92e2-45c2-9e1a-3a1087646030" path="/var/lib/kubelet/pods/93319e02-92e2-45c2-9e1a-3a1087646030/volumes"
Jul 10 00:04:48.420003 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:48.419989    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2" path="/var/lib/kubelet/pods/c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2/volumes"
Jul 10 00:04:57.090272 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.090237    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"]
Jul 10 00:04:57.090763 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.090738    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93319e02-92e2-45c2-9e1a-3a1087646030" containerName="main"
Jul 10 00:04:57.090920 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.090902    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="93319e02-92e2-45c2-9e1a-3a1087646030" containerName="main"
Jul 10 00:04:57.091002 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.090950    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2" containerName="main"
Jul 10 00:04:57.091002 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.090964    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2" containerName="main"
Jul 10 00:04:57.091109 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.091091    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4c54c13-a7e4-4f59-a3a2-9ecc9841c0f2" containerName="main"
Jul 10 00:04:57.091159 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.091108    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="93319e02-92e2-45c2-9e1a-3a1087646030" containerName="main"
Jul 10 00:04:57.094088 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.094068    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.097983 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.097954    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b0ecf1\"/\"sub-test-model-b0ecf1-kserve-self-signed-certs\""
Jul 10 00:04:57.097983 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.097963    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b0ecf1\"/\"kube-root-ca.crt\""
Jul 10 00:04:57.098179 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.097963    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b0ecf1\"/\"default-dockercfg-5ssxx\""
Jul 10 00:04:57.098179 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.098029    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b0ecf1\"/\"openshift-service-ca.crt\""
Jul 10 00:04:57.105264 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.105235    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"]
Jul 10 00:04:57.201260 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.201224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-home\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.201448 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.201280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-dshm\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.201448 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.201314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e61c48dd-e63a-4878-84ab-c37f425fe7da-tls-certs\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.201448 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.201338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8ws8\" (UniqueName: \"kubernetes.io/projected/e61c48dd-e63a-4878-84ab-c37f425fe7da-kube-api-access-n8ws8\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.201448 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.201360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-tmp-dir\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.201448 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.201418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-model-cache\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.302887 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.302851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-home\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.303071 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.302904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-dshm\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.303071 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.302927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e61c48dd-e63a-4878-84ab-c37f425fe7da-tls-certs\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.303071 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.302946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ws8\" (UniqueName: \"kubernetes.io/projected/e61c48dd-e63a-4878-84ab-c37f425fe7da-kube-api-access-n8ws8\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.303071 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.302975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-tmp-dir\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.303071 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.303002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-model-cache\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.303368 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.303347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-home\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.303411 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.303384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-tmp-dir\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.303479 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.303429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-model-cache\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.305262 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.305244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-dshm\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.305510 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.305493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e61c48dd-e63a-4878-84ab-c37f425fe7da-tls-certs\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.310183 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.310156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8ws8\" (UniqueName: \"kubernetes.io/projected/e61c48dd-e63a-4878-84ab-c37f425fe7da-kube-api-access-n8ws8\") pod \"sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") " pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.404128 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.404039    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:04:57.535428 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.535403    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"]
Jul 10 00:04:57.537333 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:04:57.537306    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode61c48dd_e63a_4878_84ab_c37f425fe7da.slice/crio-65f538e381b1e6dc0c7d629a3b9d5cf6bcf4806bce4f70556545eb4f3c77bd68 WatchSource:0}: Error finding container 65f538e381b1e6dc0c7d629a3b9d5cf6bcf4806bce4f70556545eb4f3c77bd68: Status 404 returned error can't find the container with id 65f538e381b1e6dc0c7d629a3b9d5cf6bcf4806bce4f70556545eb4f3c77bd68
Jul 10 00:04:57.939899 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.939855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n" event={"ID":"e61c48dd-e63a-4878-84ab-c37f425fe7da","Type":"ContainerStarted","Data":"4d523332c664600c32591c34e1e66bd597d9afa1d14b19e07467fb6145af77b4"}
Jul 10 00:04:57.940101 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.939904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n" event={"ID":"e61c48dd-e63a-4878-84ab-c37f425fe7da","Type":"ContainerStarted","Data":"65f538e381b1e6dc0c7d629a3b9d5cf6bcf4806bce4f70556545eb4f3c77bd68"}
Jul 10 00:04:57.955843 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:04:57.955797    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n" podStartSLOduration=0.955782849 podStartE2EDuration="955.782849ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:04:57.953952873 +0000 UTC m=+2216.097883658" watchObservedRunningTime="2026-07-10 00:04:57.955782849 +0000 UTC m=+2216.099713634"
Jul 10 00:05:07.404261 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:07.404219    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:05:07.404722 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:07.404278    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:05:07.416880 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:07.416851    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:05:08.006887 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:08.006856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:05:32.884006 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:32.883972    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"]
Jul 10 00:05:47.186242 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.186206    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"]
Jul 10 00:05:47.189853 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.189828    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.194342 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.193749    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4c394e\"/\"kube-root-ca.crt\""
Jul 10 00:05:47.194342 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.193845    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4c394e\"/\"default-dockercfg-98dhz\""
Jul 10 00:05:47.194342 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.193849    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4c394e\"/\"openshift-service-ca.crt\""
Jul 10 00:05:47.194342 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.193786    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4c394e\"/\"rate-test-model-4c394e-kserve-self-signed-certs\""
Jul 10 00:05:47.197722 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.197697    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"]
Jul 10 00:05:47.269016 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.268981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcz2f\" (UniqueName: \"kubernetes.io/projected/1a232266-9c87-442d-b746-db98a87872d0-kube-api-access-lcz2f\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.269186 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.269023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a232266-9c87-442d-b746-db98a87872d0-tls-certs\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.269186 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.269047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-dshm\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.269186 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.269095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-tmp-dir\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.269186 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.269168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-model-cache\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.269324 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.269195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-home\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.369740 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.369704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcz2f\" (UniqueName: \"kubernetes.io/projected/1a232266-9c87-442d-b746-db98a87872d0-kube-api-access-lcz2f\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.369740 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.369750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a232266-9c87-442d-b746-db98a87872d0-tls-certs\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.369968 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.369803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-dshm\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.369968 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.369854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-tmp-dir\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.369968 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.369909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-model-cache\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.369968 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.369932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-home\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.370341 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.370312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-tmp-dir\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.370341 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.370329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-home\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.370499 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.370361    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-model-cache\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.372085 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.372065    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-dshm\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.372347 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.372329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a232266-9c87-442d-b746-db98a87872d0-tls-certs\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.377029 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.377010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcz2f\" (UniqueName: \"kubernetes.io/projected/1a232266-9c87-442d-b746-db98a87872d0-kube-api-access-lcz2f\") pod \"rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") " pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.500696 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.500600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:47.639647 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.639615    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"]
Jul 10 00:05:47.640351 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:05:47.640326    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a232266_9c87_442d_b746_db98a87872d0.slice/crio-17ae57193b35a064684005c5d37610fb903af251600e022a37ba6b0ac7508592 WatchSource:0}: Error finding container 17ae57193b35a064684005c5d37610fb903af251600e022a37ba6b0ac7508592: Status 404 returned error can't find the container with id 17ae57193b35a064684005c5d37610fb903af251600e022a37ba6b0ac7508592
Jul 10 00:05:47.904906 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.904841    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n" podUID="e61c48dd-e63a-4878-84ab-c37f425fe7da" containerName="main" containerID="cri-o://4d523332c664600c32591c34e1e66bd597d9afa1d14b19e07467fb6145af77b4" gracePeriod=45
Jul 10 00:05:47.996844 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:47.996791    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n" podUID="e61c48dd-e63a-4878-84ab-c37f425fe7da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 10 00:05:48.153355 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.153328    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:05:48.217729 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.217602    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r" event={"ID":"1a232266-9c87-442d-b746-db98a87872d0","Type":"ContainerStarted","Data":"6ea441bf99d5b824f2d41a292eed93c58caef7771fd2be12d638181a1ea53c6f"}
Jul 10 00:05:48.217729 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.217678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r" event={"ID":"1a232266-9c87-442d-b746-db98a87872d0","Type":"ContainerStarted","Data":"17ae57193b35a064684005c5d37610fb903af251600e022a37ba6b0ac7508592"}
Jul 10 00:05:48.218901 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.218874    2580 generic.go:356] "Generic (PLEG): container finished" podID="e61c48dd-e63a-4878-84ab-c37f425fe7da" containerID="4d523332c664600c32591c34e1e66bd597d9afa1d14b19e07467fb6145af77b4" exitCode=0
Jul 10 00:05:48.219019 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.218908    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"
Jul 10 00:05:48.219019 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.218957    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n" event={"ID":"e61c48dd-e63a-4878-84ab-c37f425fe7da","Type":"ContainerDied","Data":"4d523332c664600c32591c34e1e66bd597d9afa1d14b19e07467fb6145af77b4"}
Jul 10 00:05:48.219019 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.218996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n" event={"ID":"e61c48dd-e63a-4878-84ab-c37f425fe7da","Type":"ContainerDied","Data":"65f538e381b1e6dc0c7d629a3b9d5cf6bcf4806bce4f70556545eb4f3c77bd68"}
Jul 10 00:05:48.219019 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.219017    2580 scope.go:117] "RemoveContainer" containerID="4d523332c664600c32591c34e1e66bd597d9afa1d14b19e07467fb6145af77b4"
Jul 10 00:05:48.227928 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.227833    2580 scope.go:117] "RemoveContainer" containerID="4d523332c664600c32591c34e1e66bd597d9afa1d14b19e07467fb6145af77b4"
Jul 10 00:05:48.228228 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:05:48.228199    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d523332c664600c32591c34e1e66bd597d9afa1d14b19e07467fb6145af77b4\": container with ID starting with 4d523332c664600c32591c34e1e66bd597d9afa1d14b19e07467fb6145af77b4 not found: ID does not exist" containerID="4d523332c664600c32591c34e1e66bd597d9afa1d14b19e07467fb6145af77b4"
Jul 10 00:05:48.228364 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.228239    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d523332c664600c32591c34e1e66bd597d9afa1d14b19e07467fb6145af77b4"} err="failed to get container status \"4d523332c664600c32591c34e1e66bd597d9afa1d14b19e07467fb6145af77b4\": rpc error: code = NotFound desc = could not find container \"4d523332c664600c32591c34e1e66bd597d9afa1d14b19e07467fb6145af77b4\": container with ID starting with 4d523332c664600c32591c34e1e66bd597d9afa1d14b19e07467fb6145af77b4 not found: ID does not exist"
Jul 10 00:05:48.233079 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.233033    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r" podStartSLOduration=1.233019634 podStartE2EDuration="1.233019634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:05:48.232393425 +0000 UTC m=+2266.376324213" watchObservedRunningTime="2026-07-10 00:05:48.233019634 +0000 UTC m=+2266.376950460"
Jul 10 00:05:48.276416 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.276379    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-home\") pod \"e61c48dd-e63a-4878-84ab-c37f425fe7da\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") "
Jul 10 00:05:48.276609 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.276433    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8ws8\" (UniqueName: \"kubernetes.io/projected/e61c48dd-e63a-4878-84ab-c37f425fe7da-kube-api-access-n8ws8\") pod \"e61c48dd-e63a-4878-84ab-c37f425fe7da\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") "
Jul 10 00:05:48.276609 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.276456    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e61c48dd-e63a-4878-84ab-c37f425fe7da-tls-certs\") pod \"e61c48dd-e63a-4878-84ab-c37f425fe7da\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") "
Jul 10 00:05:48.276609 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.276488    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-tmp-dir\") pod \"e61c48dd-e63a-4878-84ab-c37f425fe7da\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") "
Jul 10 00:05:48.276609 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.276535    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-dshm\") pod \"e61c48dd-e63a-4878-84ab-c37f425fe7da\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") "
Jul 10 00:05:48.276868 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.276714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-model-cache\") pod \"e61c48dd-e63a-4878-84ab-c37f425fe7da\" (UID: \"e61c48dd-e63a-4878-84ab-c37f425fe7da\") "
Jul 10 00:05:48.276868 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.276783    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-home" (OuterVolumeSpecName: "home") pod "e61c48dd-e63a-4878-84ab-c37f425fe7da" (UID: "e61c48dd-e63a-4878-84ab-c37f425fe7da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:05:48.276868 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.276829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e61c48dd-e63a-4878-84ab-c37f425fe7da" (UID: "e61c48dd-e63a-4878-84ab-c37f425fe7da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:05:48.277038 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.276948    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-model-cache" (OuterVolumeSpecName: "model-cache") pod "e61c48dd-e63a-4878-84ab-c37f425fe7da" (UID: "e61c48dd-e63a-4878-84ab-c37f425fe7da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:05:48.277097 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.277084    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-home\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:05:48.277153 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.277096    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-tmp-dir\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:05:48.277153 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.277106    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-model-cache\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:05:48.278596 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.278575    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61c48dd-e63a-4878-84ab-c37f425fe7da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e61c48dd-e63a-4878-84ab-c37f425fe7da" (UID: "e61c48dd-e63a-4878-84ab-c37f425fe7da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 00:05:48.278700 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.278604    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-dshm" (OuterVolumeSpecName: "dshm") pod "e61c48dd-e63a-4878-84ab-c37f425fe7da" (UID: "e61c48dd-e63a-4878-84ab-c37f425fe7da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:05:48.278700 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.278627    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61c48dd-e63a-4878-84ab-c37f425fe7da-kube-api-access-n8ws8" (OuterVolumeSpecName: "kube-api-access-n8ws8") pod "e61c48dd-e63a-4878-84ab-c37f425fe7da" (UID: "e61c48dd-e63a-4878-84ab-c37f425fe7da"). InnerVolumeSpecName "kube-api-access-n8ws8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:05:48.378321 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.378288    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e61c48dd-e63a-4878-84ab-c37f425fe7da-dshm\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:05:48.378321 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.378321    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8ws8\" (UniqueName: \"kubernetes.io/projected/e61c48dd-e63a-4878-84ab-c37f425fe7da-kube-api-access-n8ws8\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:05:48.378765 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.378336    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e61c48dd-e63a-4878-84ab-c37f425fe7da-tls-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:05:48.536125 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.536095    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"]
Jul 10 00:05:48.538083 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:48.538062    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/sub-test-model-b0ecf1-kserve-9f79f75b9-4kd4n"]
Jul 10 00:05:50.419207 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:50.419173    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e61c48dd-e63a-4878-84ab-c37f425fe7da" path="/var/lib/kubelet/pods/e61c48dd-e63a-4878-84ab-c37f425fe7da/volumes"
Jul 10 00:05:57.501705 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:57.501640    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:57.502154 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:57.501726    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:57.514254 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:57.514223    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:05:58.286163 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:05:58.286131    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:06:03.395966 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.395930    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"]
Jul 10 00:06:03.396372 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.396338    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61c48dd-e63a-4878-84ab-c37f425fe7da" containerName="main"
Jul 10 00:06:03.396372 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.396349    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61c48dd-e63a-4878-84ab-c37f425fe7da" containerName="main"
Jul 10 00:06:03.396453 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.396403    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e61c48dd-e63a-4878-84ab-c37f425fe7da" containerName="main"
Jul 10 00:06:03.400470 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.400454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.405485 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.405465    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b0ecf1\"/\"kube-root-ca.crt\""
Jul 10 00:06:03.406426 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.406402    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b0ecf1\"/\"default-dockercfg-5ssxx\""
Jul 10 00:06:03.406684 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.406662    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b0ecf1\"/\"openshift-service-ca.crt\""
Jul 10 00:06:03.406800 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.406721    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b0ecf1\"/\"rate-test-model-b0ecf1-kserve-self-signed-certs\""
Jul 10 00:06:03.414834 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.414812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-dshm\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.414949 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.414845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-home\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.414949 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.414892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-model-cache\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.415036 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.414954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qckh9\" (UniqueName: \"kubernetes.io/projected/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-kube-api-access-qckh9\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.415036 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.414993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-tmp-dir\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.415115 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.415047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-tls-certs\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.419050 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.419029    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"]
Jul 10 00:06:03.515596 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.515557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-model-cache\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.515596 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.515599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qckh9\" (UniqueName: \"kubernetes.io/projected/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-kube-api-access-qckh9\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.515870 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.515621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-tmp-dir\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.515870 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.515697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-tls-certs\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.515870 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.515745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-dshm\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.515870 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.515766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-home\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.516097 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.516043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-model-cache\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.516097 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.516073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-tmp-dir\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.516203 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.516186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-home\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.518113 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.518091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-dshm\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.518354 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.518332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-tls-certs\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.529529 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.529502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qckh9\" (UniqueName: \"kubernetes.io/projected/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-kube-api-access-qckh9\") pod \"rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") " pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.710122 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.710029    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:03.853749 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:03.853722    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"]
Jul 10 00:06:03.855436 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:06:03.855410    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90d8466f_9e6b_4b67_b7b6_9bd2a3b887cb.slice/crio-64b62d80855aefb5e9f76268e9513dfe972efe2a87d9cf151beb5ade6e46fd18 WatchSource:0}: Error finding container 64b62d80855aefb5e9f76268e9513dfe972efe2a87d9cf151beb5ade6e46fd18: Status 404 returned error can't find the container with id 64b62d80855aefb5e9f76268e9513dfe972efe2a87d9cf151beb5ade6e46fd18
Jul 10 00:06:04.309110 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:04.309070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg" event={"ID":"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb","Type":"ContainerStarted","Data":"058d696221aab4e2e8682d3e2e98c5aa8b217e2bb0afa8da622fbe0ff5531f82"}
Jul 10 00:06:04.309317 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:04.309120    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg" event={"ID":"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb","Type":"ContainerStarted","Data":"64b62d80855aefb5e9f76268e9513dfe972efe2a87d9cf151beb5ade6e46fd18"}
Jul 10 00:06:04.327554 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:04.327502    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg" podStartSLOduration=1.327486527 podStartE2EDuration="1.327486527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:06:04.32424816 +0000 UTC m=+2282.468178949" watchObservedRunningTime="2026-07-10 00:06:04.327486527 +0000 UTC m=+2282.471417313"
Jul 10 00:06:13.710185 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:13.710132    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:13.710601 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:13.710200    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:13.722522 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:13.722493    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:14.381374 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:14.381344    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:06:54.179066 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:54.179029    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"]
Jul 10 00:06:55.197281 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:06:55.197246    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"]
Jul 10 00:07:09.201754 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.201710    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r" podUID="1a232266-9c87-442d-b746-db98a87872d0" containerName="main" containerID="cri-o://6ea441bf99d5b824f2d41a292eed93c58caef7771fd2be12d638181a1ea53c6f" gracePeriod=45
Jul 10 00:07:09.468923 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.468893    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:07:09.509096 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.509055    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-home\") pod \"1a232266-9c87-442d-b746-db98a87872d0\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") "
Jul 10 00:07:09.509255 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.509106    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-dshm\") pod \"1a232266-9c87-442d-b746-db98a87872d0\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") "
Jul 10 00:07:09.509255 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.509191    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-model-cache\") pod \"1a232266-9c87-442d-b746-db98a87872d0\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") "
Jul 10 00:07:09.509352 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.509259    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcz2f\" (UniqueName: \"kubernetes.io/projected/1a232266-9c87-442d-b746-db98a87872d0-kube-api-access-lcz2f\") pod \"1a232266-9c87-442d-b746-db98a87872d0\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") "
Jul 10 00:07:09.509352 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.509293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-tmp-dir\") pod \"1a232266-9c87-442d-b746-db98a87872d0\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") "
Jul 10 00:07:09.509352 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.509327    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a232266-9c87-442d-b746-db98a87872d0-tls-certs\") pod \"1a232266-9c87-442d-b746-db98a87872d0\" (UID: \"1a232266-9c87-442d-b746-db98a87872d0\") "
Jul 10 00:07:09.509352 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.509324    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-home" (OuterVolumeSpecName: "home") pod "1a232266-9c87-442d-b746-db98a87872d0" (UID: "1a232266-9c87-442d-b746-db98a87872d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:09.509560 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.509394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a232266-9c87-442d-b746-db98a87872d0" (UID: "1a232266-9c87-442d-b746-db98a87872d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:09.509560 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.509497    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a232266-9c87-442d-b746-db98a87872d0" (UID: "1a232266-9c87-442d-b746-db98a87872d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:09.509698 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.509681    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-tmp-dir\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:09.509776 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.509699    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-home\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:09.509776 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.509712    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-model-cache\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:09.511524 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.511494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a232266-9c87-442d-b746-db98a87872d0-kube-api-access-lcz2f" (OuterVolumeSpecName: "kube-api-access-lcz2f") pod "1a232266-9c87-442d-b746-db98a87872d0" (UID: "1a232266-9c87-442d-b746-db98a87872d0"). InnerVolumeSpecName "kube-api-access-lcz2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:07:09.511823 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.511797    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-dshm" (OuterVolumeSpecName: "dshm") pod "1a232266-9c87-442d-b746-db98a87872d0" (UID: "1a232266-9c87-442d-b746-db98a87872d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:09.511921 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.511818    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a232266-9c87-442d-b746-db98a87872d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a232266-9c87-442d-b746-db98a87872d0" (UID: "1a232266-9c87-442d-b746-db98a87872d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 00:07:09.611125 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.611092    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcz2f\" (UniqueName: \"kubernetes.io/projected/1a232266-9c87-442d-b746-db98a87872d0-kube-api-access-lcz2f\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:09.611125 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.611123    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a232266-9c87-442d-b746-db98a87872d0-tls-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:09.611336 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.611136    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a232266-9c87-442d-b746-db98a87872d0-dshm\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:09.660144 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.660110    2580 generic.go:356] "Generic (PLEG): container finished" podID="1a232266-9c87-442d-b746-db98a87872d0" containerID="6ea441bf99d5b824f2d41a292eed93c58caef7771fd2be12d638181a1ea53c6f" exitCode=0
Jul 10 00:07:09.660340 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.660155    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"
Jul 10 00:07:09.660340 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.660198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r" event={"ID":"1a232266-9c87-442d-b746-db98a87872d0","Type":"ContainerDied","Data":"6ea441bf99d5b824f2d41a292eed93c58caef7771fd2be12d638181a1ea53c6f"}
Jul 10 00:07:09.660340 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.660242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r" event={"ID":"1a232266-9c87-442d-b746-db98a87872d0","Type":"ContainerDied","Data":"17ae57193b35a064684005c5d37610fb903af251600e022a37ba6b0ac7508592"}
Jul 10 00:07:09.660340 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.660255    2580 scope.go:117] "RemoveContainer" containerID="6ea441bf99d5b824f2d41a292eed93c58caef7771fd2be12d638181a1ea53c6f"
Jul 10 00:07:09.670247 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.670231    2580 scope.go:117] "RemoveContainer" containerID="6ea441bf99d5b824f2d41a292eed93c58caef7771fd2be12d638181a1ea53c6f"
Jul 10 00:07:09.670522 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:07:09.670505    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea441bf99d5b824f2d41a292eed93c58caef7771fd2be12d638181a1ea53c6f\": container with ID starting with 6ea441bf99d5b824f2d41a292eed93c58caef7771fd2be12d638181a1ea53c6f not found: ID does not exist" containerID="6ea441bf99d5b824f2d41a292eed93c58caef7771fd2be12d638181a1ea53c6f"
Jul 10 00:07:09.670568 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.670541    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea441bf99d5b824f2d41a292eed93c58caef7771fd2be12d638181a1ea53c6f"} err="failed to get container status \"6ea441bf99d5b824f2d41a292eed93c58caef7771fd2be12d638181a1ea53c6f\": rpc error: code = NotFound desc = could not find container \"6ea441bf99d5b824f2d41a292eed93c58caef7771fd2be12d638181a1ea53c6f\": container with ID starting with 6ea441bf99d5b824f2d41a292eed93c58caef7771fd2be12d638181a1ea53c6f not found: ID does not exist"
Jul 10 00:07:09.680721 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.680694    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"]
Jul 10 00:07:09.684759 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:09.684738    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/rate-test-model-4c394e-kserve-7fc6cc88cf-xdw4r"]
Jul 10 00:07:10.219774 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.219734    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg" podUID="90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb" containerName="main" containerID="cri-o://058d696221aab4e2e8682d3e2e98c5aa8b217e2bb0afa8da622fbe0ff5531f82" gracePeriod=45
Jul 10 00:07:10.420864 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.420832    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a232266-9c87-442d-b746-db98a87872d0" path="/var/lib/kubelet/pods/1a232266-9c87-442d-b746-db98a87872d0/volumes"
Jul 10 00:07:10.478873 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.478812    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:07:10.520448 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.520417    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-home\") pod \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") "
Jul 10 00:07:10.520628 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.520464    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-model-cache\") pod \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") "
Jul 10 00:07:10.520628 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.520496    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-dshm\") pod \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") "
Jul 10 00:07:10.520628 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.520538    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-tmp-dir\") pod \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") "
Jul 10 00:07:10.520628 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.520554    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-tls-certs\") pod \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") "
Jul 10 00:07:10.520628 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.520623    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qckh9\" (UniqueName: \"kubernetes.io/projected/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-kube-api-access-qckh9\") pod \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\" (UID: \"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb\") "
Jul 10 00:07:10.520932 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.520761    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-home" (OuterVolumeSpecName: "home") pod "90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb" (UID: "90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:10.520932 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.520753    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb" (UID: "90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:10.520932 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.520852    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb" (UID: "90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:10.520932 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.520890    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-home\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:10.520932 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.520909    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-model-cache\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:10.522759 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.522728    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-dshm" (OuterVolumeSpecName: "dshm") pod "90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb" (UID: "90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:10.522855 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.522789    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-kube-api-access-qckh9" (OuterVolumeSpecName: "kube-api-access-qckh9") pod "90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb" (UID: "90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb"). InnerVolumeSpecName "kube-api-access-qckh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:07:10.522896 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.522848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb" (UID: "90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 00:07:10.621698 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.621631    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-dshm\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:10.621698 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.621691    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-tmp-dir\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:10.621698 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.621702    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-tls-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:10.621942 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.621712    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qckh9\" (UniqueName: \"kubernetes.io/projected/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb-kube-api-access-qckh9\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:10.667473 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.667439    2580 generic.go:356] "Generic (PLEG): container finished" podID="90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb" containerID="058d696221aab4e2e8682d3e2e98c5aa8b217e2bb0afa8da622fbe0ff5531f82" exitCode=0
Jul 10 00:07:10.667673 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.667492    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"
Jul 10 00:07:10.667673 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.667500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg" event={"ID":"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb","Type":"ContainerDied","Data":"058d696221aab4e2e8682d3e2e98c5aa8b217e2bb0afa8da622fbe0ff5531f82"}
Jul 10 00:07:10.667673 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.667529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg" event={"ID":"90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb","Type":"ContainerDied","Data":"64b62d80855aefb5e9f76268e9513dfe972efe2a87d9cf151beb5ade6e46fd18"}
Jul 10 00:07:10.667673 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.667546    2580 scope.go:117] "RemoveContainer" containerID="058d696221aab4e2e8682d3e2e98c5aa8b217e2bb0afa8da622fbe0ff5531f82"
Jul 10 00:07:10.677142 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.677122    2580 scope.go:117] "RemoveContainer" containerID="058d696221aab4e2e8682d3e2e98c5aa8b217e2bb0afa8da622fbe0ff5531f82"
Jul 10 00:07:10.677386 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:07:10.677366    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058d696221aab4e2e8682d3e2e98c5aa8b217e2bb0afa8da622fbe0ff5531f82\": container with ID starting with 058d696221aab4e2e8682d3e2e98c5aa8b217e2bb0afa8da622fbe0ff5531f82 not found: ID does not exist" containerID="058d696221aab4e2e8682d3e2e98c5aa8b217e2bb0afa8da622fbe0ff5531f82"
Jul 10 00:07:10.677434 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.677396    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058d696221aab4e2e8682d3e2e98c5aa8b217e2bb0afa8da622fbe0ff5531f82"} err="failed to get container status \"058d696221aab4e2e8682d3e2e98c5aa8b217e2bb0afa8da622fbe0ff5531f82\": rpc error: code = NotFound desc = could not find container \"058d696221aab4e2e8682d3e2e98c5aa8b217e2bb0afa8da622fbe0ff5531f82\": container with ID starting with 058d696221aab4e2e8682d3e2e98c5aa8b217e2bb0afa8da622fbe0ff5531f82 not found: ID does not exist"
Jul 10 00:07:10.687639 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.687606    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"]
Jul 10 00:07:10.689737 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.689702    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b0ecf1/rate-test-model-b0ecf1-kserve-76c759f7f-sh4kg"]
Jul 10 00:07:10.797524 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.797486    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7224-140222186908528"]
Jul 10 00:07:10.798098 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.798081    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a232266-9c87-442d-b746-db98a87872d0" containerName="main"
Jul 10 00:07:10.798161 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.798101    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a232266-9c87-442d-b746-db98a87872d0" containerName="main"
Jul 10 00:07:10.798161 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.798122    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb" containerName="main"
Jul 10 00:07:10.798161 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.798127    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb" containerName="main"
Jul 10 00:07:10.798254 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.798210    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb" containerName="main"
Jul 10 00:07:10.798254 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.798220    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a232266-9c87-442d-b746-db98a87872d0" containerName="main"
Jul 10 00:07:10.802450 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.802434    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186908528"
Jul 10 00:07:10.805627 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.805599    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-c9bdn\""
Jul 10 00:07:10.807040 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.807018    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7224-140222186908528"]
Jul 10 00:07:10.924521 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:10.924480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmbw5\" (UniqueName: \"kubernetes.io/projected/5e2d9227-a0e7-4f87-b5a9-d045c4b793ff-kube-api-access-pmbw5\") pod \"test-curl-7224-140222186908528\" (UID: \"5e2d9227-a0e7-4f87-b5a9-d045c4b793ff\") " pod="openshift-ingress/test-curl-7224-140222186908528"
Jul 10 00:07:11.025121 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:11.025089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmbw5\" (UniqueName: \"kubernetes.io/projected/5e2d9227-a0e7-4f87-b5a9-d045c4b793ff-kube-api-access-pmbw5\") pod \"test-curl-7224-140222186908528\" (UID: \"5e2d9227-a0e7-4f87-b5a9-d045c4b793ff\") " pod="openshift-ingress/test-curl-7224-140222186908528"
Jul 10 00:07:11.032320 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:11.032292    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmbw5\" (UniqueName: \"kubernetes.io/projected/5e2d9227-a0e7-4f87-b5a9-d045c4b793ff-kube-api-access-pmbw5\") pod \"test-curl-7224-140222186908528\" (UID: \"5e2d9227-a0e7-4f87-b5a9-d045c4b793ff\") " pod="openshift-ingress/test-curl-7224-140222186908528"
Jul 10 00:07:11.113070 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:11.112979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186908528"
Jul 10 00:07:11.246493 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:11.246455    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7224-140222186908528"]
Jul 10 00:07:11.249125 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:07:11.249092    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e2d9227_a0e7_4f87_b5a9_d045c4b793ff.slice/crio-925a87b506d0181d8bf9a5a0107c8e8e2f49dbb42d0618328bf6b653f7ffd428 WatchSource:0}: Error finding container 925a87b506d0181d8bf9a5a0107c8e8e2f49dbb42d0618328bf6b653f7ffd428: Status 404 returned error can't find the container with id 925a87b506d0181d8bf9a5a0107c8e8e2f49dbb42d0618328bf6b653f7ffd428
Jul 10 00:07:11.250510 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:11.250482    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 00:07:11.674457 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:11.674420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7224-140222186908528" event={"ID":"5e2d9227-a0e7-4f87-b5a9-d045c4b793ff","Type":"ContainerStarted","Data":"925a87b506d0181d8bf9a5a0107c8e8e2f49dbb42d0618328bf6b653f7ffd428"}
Jul 10 00:07:12.419431 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:12.419389    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb" path="/var/lib/kubelet/pods/90d8466f-9e6b-4b67-b7b6-9bd2a3b887cb/volumes"
Jul 10 00:07:12.682441 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:12.682350    2580 generic.go:356] "Generic (PLEG): container finished" podID="5e2d9227-a0e7-4f87-b5a9-d045c4b793ff" containerID="d0d92e097c8f83e29a106a9f0751e1d0917f59bb1245a84647ca044c9a3e5124" exitCode=0
Jul 10 00:07:12.682441 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:12.682393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7224-140222186908528" event={"ID":"5e2d9227-a0e7-4f87-b5a9-d045c4b793ff","Type":"ContainerDied","Data":"d0d92e097c8f83e29a106a9f0751e1d0917f59bb1245a84647ca044c9a3e5124"}
Jul 10 00:07:13.828266 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:13.828240    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186908528"
Jul 10 00:07:13.862631 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:13.862595    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7224-140222186908528_5e2d9227-a0e7-4f87-b5a9-d045c4b793ff/test-curl-7224-140222186908528/0.log"
Jul 10 00:07:13.885233 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:13.885198    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7224-140222186908528"]
Jul 10 00:07:13.888703 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:13.888678    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7224-140222186908528"]
Jul 10 00:07:13.950597 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:13.950507    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmbw5\" (UniqueName: \"kubernetes.io/projected/5e2d9227-a0e7-4f87-b5a9-d045c4b793ff-kube-api-access-pmbw5\") pod \"5e2d9227-a0e7-4f87-b5a9-d045c4b793ff\" (UID: \"5e2d9227-a0e7-4f87-b5a9-d045c4b793ff\") "
Jul 10 00:07:13.952696 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:13.952671    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2d9227-a0e7-4f87-b5a9-d045c4b793ff-kube-api-access-pmbw5" (OuterVolumeSpecName: "kube-api-access-pmbw5") pod "5e2d9227-a0e7-4f87-b5a9-d045c4b793ff" (UID: "5e2d9227-a0e7-4f87-b5a9-d045c4b793ff"). InnerVolumeSpecName "kube-api-access-pmbw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:07:14.051613 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:14.051585    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmbw5\" (UniqueName: \"kubernetes.io/projected/5e2d9227-a0e7-4f87-b5a9-d045c4b793ff-kube-api-access-pmbw5\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:14.418985 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:14.418953    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e2d9227-a0e7-4f87-b5a9-d045c4b793ff" path="/var/lib/kubelet/pods/5e2d9227-a0e7-4f87-b5a9-d045c4b793ff/volumes"
Jul 10 00:07:14.694335 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:14.694240    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186908528"
Jul 10 00:07:14.694335 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:14.694259    2580 scope.go:117] "RemoveContainer" containerID="d0d92e097c8f83e29a106a9f0751e1d0917f59bb1245a84647ca044c9a3e5124"
Jul 10 00:07:20.437731 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:20.437681    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"]
Jul 10 00:07:20.438208 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:20.437935    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy" containerID="cri-o://50e5a8e1d6f2ea520db1ca24db28674c63174e3ce373ee84023d0dae6b8d7084" gracePeriod=30
Jul 10 00:07:21.452074 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:21.452034    2580 patch_prober.go:28] interesting pod/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 10 00:07:21.452439 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:21.452093    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 00:07:25.707191 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.707130    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:07:25.758543 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.758513    2580 generic.go:356] "Generic (PLEG): container finished" podID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerID="50e5a8e1d6f2ea520db1ca24db28674c63174e3ce373ee84023d0dae6b8d7084" exitCode=0
Jul 10 00:07:25.758700 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.758563    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"
Jul 10 00:07:25.758700 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.758566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" event={"ID":"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688","Type":"ContainerDied","Data":"50e5a8e1d6f2ea520db1ca24db28674c63174e3ce373ee84023d0dae6b8d7084"}
Jul 10 00:07:25.758788 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.758712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn" event={"ID":"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688","Type":"ContainerDied","Data":"3ac33aebbce33efd8bd2cb42170f261b615345e38ce139bd7e98fd8c8d186e85"}
Jul 10 00:07:25.758788 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.758743    2580 scope.go:117] "RemoveContainer" containerID="50e5a8e1d6f2ea520db1ca24db28674c63174e3ce373ee84023d0dae6b8d7084"
Jul 10 00:07:25.770059 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.769361    2580 scope.go:117] "RemoveContainer" containerID="cd176b6ebdaede63e1ea0a12067f8407d9228673c5cde2ac3a361e41c25cf279"
Jul 10 00:07:25.778347 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.778317    2580 scope.go:117] "RemoveContainer" containerID="50e5a8e1d6f2ea520db1ca24db28674c63174e3ce373ee84023d0dae6b8d7084"
Jul 10 00:07:25.778604 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:07:25.778581    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e5a8e1d6f2ea520db1ca24db28674c63174e3ce373ee84023d0dae6b8d7084\": container with ID starting with 50e5a8e1d6f2ea520db1ca24db28674c63174e3ce373ee84023d0dae6b8d7084 not found: ID does not exist" containerID="50e5a8e1d6f2ea520db1ca24db28674c63174e3ce373ee84023d0dae6b8d7084"
Jul 10 00:07:25.778664 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.778613    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e5a8e1d6f2ea520db1ca24db28674c63174e3ce373ee84023d0dae6b8d7084"} err="failed to get container status \"50e5a8e1d6f2ea520db1ca24db28674c63174e3ce373ee84023d0dae6b8d7084\": rpc error: code = NotFound desc = could not find container \"50e5a8e1d6f2ea520db1ca24db28674c63174e3ce373ee84023d0dae6b8d7084\": container with ID starting with 50e5a8e1d6f2ea520db1ca24db28674c63174e3ce373ee84023d0dae6b8d7084 not found: ID does not exist"
Jul 10 00:07:25.778664 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.778633    2580 scope.go:117] "RemoveContainer" containerID="cd176b6ebdaede63e1ea0a12067f8407d9228673c5cde2ac3a361e41c25cf279"
Jul 10 00:07:25.778900 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:07:25.778882    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd176b6ebdaede63e1ea0a12067f8407d9228673c5cde2ac3a361e41c25cf279\": container with ID starting with cd176b6ebdaede63e1ea0a12067f8407d9228673c5cde2ac3a361e41c25cf279 not found: ID does not exist" containerID="cd176b6ebdaede63e1ea0a12067f8407d9228673c5cde2ac3a361e41c25cf279"
Jul 10 00:07:25.778959 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.778922    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd176b6ebdaede63e1ea0a12067f8407d9228673c5cde2ac3a361e41c25cf279"} err="failed to get container status \"cd176b6ebdaede63e1ea0a12067f8407d9228673c5cde2ac3a361e41c25cf279\": rpc error: code = NotFound desc = could not find container \"cd176b6ebdaede63e1ea0a12067f8407d9228673c5cde2ac3a361e41c25cf279\": container with ID starting with cd176b6ebdaede63e1ea0a12067f8407d9228673c5cde2ac3a361e41c25cf279 not found: ID does not exist"
Jul 10 00:07:25.858352 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.858322    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-workload-socket\") pod \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") "
Jul 10 00:07:25.858553 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.858376    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-token\") pod \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") "
Jul 10 00:07:25.858553 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.858409    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-data\") pod \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") "
Jul 10 00:07:25.858553 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.858438    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-credential-socket\") pod \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") "
Jul 10 00:07:25.858553 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.858470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istiod-ca-cert\") pod \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") "
Jul 10 00:07:25.858812 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.858601    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" (UID: "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:25.858812 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.858620    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-envoy\") pod \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") "
Jul 10 00:07:25.858812 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.858696    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-workload-certs\") pod \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") "
Jul 10 00:07:25.858812 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.858760    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-podinfo\") pod \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") "
Jul 10 00:07:25.858812 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.858789    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" (UID: "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:25.859053 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.858796    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76q9w\" (UniqueName: \"kubernetes.io/projected/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-kube-api-access-76q9w\") pod \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\" (UID: \"5cc569f4-ee41-41ee-a4ea-9d4e06cb6688\") "
Jul 10 00:07:25.859053 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.858911    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" (UID: "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 00:07:25.859169 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.859053    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" (UID: "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:25.859245 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.859222    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-workload-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:25.859310 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.859257    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-credential-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:25.859310 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.859271    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istiod-ca-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:25.859310 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.859284    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-workload-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:25.859495 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.859470    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-data" (OuterVolumeSpecName: "istio-data") pod "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" (UID: "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:25.860905 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.860881    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" (UID: "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:25.860999 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.860916    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-kube-api-access-76q9w" (OuterVolumeSpecName: "kube-api-access-76q9w") pod "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" (UID: "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688"). InnerVolumeSpecName "kube-api-access-76q9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:07:25.861185 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.861168    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" (UID: "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 10 00:07:25.861247 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.861190    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-token" (OuterVolumeSpecName: "istio-token") pod "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" (UID: "5cc569f4-ee41-41ee-a4ea-9d4e06cb6688"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:07:25.960720 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.960613    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-token\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:25.960720 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.960644    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-data\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:25.960720 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.960673    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-envoy\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:25.960720 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.960684    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-istio-podinfo\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:25.960720 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:25.960696    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76q9w\" (UniqueName: \"kubernetes.io/projected/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688-kube-api-access-76q9w\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:26.081186 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:26.081141    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"]
Jul 10 00:07:26.085223 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:26.085196    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-4c394e-openshift-default-646f4857b7-pwvmn"]
Jul 10 00:07:26.419568 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:26.419534    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" path="/var/lib/kubelet/pods/5cc569f4-ee41-41ee-a4ea-9d4e06cb6688/volumes"
Jul 10 00:07:27.291549 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:27.291512    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"]
Jul 10 00:07:27.292010 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:27.291814    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy" containerID="cri-o://1b14cce6276cf092657f9b26dd1ae6b6e031560c54f1ea4ad2dec73737f45db7" gracePeriod=30
Jul 10 00:07:27.933362 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:27.933326    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b55548785-pmhm9_85f4e702-49a2-42d8-a08f-63f9242654c2/authorino/0.log"
Jul 10 00:07:29.134542 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:29.134506    2580 patch_prober.go:28] interesting pod/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 10 00:07:29.134927 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:29.134562    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 00:07:32.548484 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.548453    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:07:32.591976 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.591899    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-l9vqc_686df5b5-af27-4125-8c70-5b8f0a52afdf/manager/2.log"
Jul 10 00:07:32.703818 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.703784    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-nmfvk_0719fe1b-21a5-4adc-b802-cd591b01cc17/manager/0.log"
Jul 10 00:07:32.724322 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.724292    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm5q5\" (UniqueName: \"kubernetes.io/projected/dac4d79e-ebfe-453a-8a11-cd3082264301-kube-api-access-qm5q5\") pod \"dac4d79e-ebfe-453a-8a11-cd3082264301\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") "
Jul 10 00:07:32.724504 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.724357    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac4d79e-ebfe-453a-8a11-cd3082264301-istiod-ca-cert\") pod \"dac4d79e-ebfe-453a-8a11-cd3082264301\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") "
Jul 10 00:07:32.724504 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.724457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-credential-socket\") pod \"dac4d79e-ebfe-453a-8a11-cd3082264301\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") "
Jul 10 00:07:32.724504 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.724492    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-workload-socket\") pod \"dac4d79e-ebfe-453a-8a11-cd3082264301\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") "
Jul 10 00:07:32.724701 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.724540    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-podinfo\") pod \"dac4d79e-ebfe-453a-8a11-cd3082264301\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") "
Jul 10 00:07:32.724701 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.724567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-data\") pod \"dac4d79e-ebfe-453a-8a11-cd3082264301\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") "
Jul 10 00:07:32.724701 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.724585    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-envoy\") pod \"dac4d79e-ebfe-453a-8a11-cd3082264301\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") "
Jul 10 00:07:32.724855 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.724720    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dac4d79e-ebfe-453a-8a11-cd3082264301" (UID: "dac4d79e-ebfe-453a-8a11-cd3082264301"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:32.724855 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.724770    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-token\") pod \"dac4d79e-ebfe-453a-8a11-cd3082264301\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") "
Jul 10 00:07:32.724855 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.724850    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-workload-certs\") pod \"dac4d79e-ebfe-453a-8a11-cd3082264301\" (UID: \"dac4d79e-ebfe-453a-8a11-cd3082264301\") "
Jul 10 00:07:32.725026 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.724852    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac4d79e-ebfe-453a-8a11-cd3082264301-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dac4d79e-ebfe-453a-8a11-cd3082264301" (UID: "dac4d79e-ebfe-453a-8a11-cd3082264301"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 00:07:32.725026 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.724881    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dac4d79e-ebfe-453a-8a11-cd3082264301" (UID: "dac4d79e-ebfe-453a-8a11-cd3082264301"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:32.725142 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.725125    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-credential-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:32.725207 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.725145    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-workload-socket\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:32.725207 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.725159    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac4d79e-ebfe-453a-8a11-cd3082264301-istiod-ca-cert\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:32.725207 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.725192    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dac4d79e-ebfe-453a-8a11-cd3082264301" (UID: "dac4d79e-ebfe-453a-8a11-cd3082264301"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:32.725497 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.725468    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-data" (OuterVolumeSpecName: "istio-data") pod "dac4d79e-ebfe-453a-8a11-cd3082264301" (UID: "dac4d79e-ebfe-453a-8a11-cd3082264301"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:32.726676 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.726609    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac4d79e-ebfe-453a-8a11-cd3082264301-kube-api-access-qm5q5" (OuterVolumeSpecName: "kube-api-access-qm5q5") pod "dac4d79e-ebfe-453a-8a11-cd3082264301" (UID: "dac4d79e-ebfe-453a-8a11-cd3082264301"). InnerVolumeSpecName "kube-api-access-qm5q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:07:32.726676 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.726614    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dac4d79e-ebfe-453a-8a11-cd3082264301" (UID: "dac4d79e-ebfe-453a-8a11-cd3082264301"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 10 00:07:32.726811 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.726696    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dac4d79e-ebfe-453a-8a11-cd3082264301" (UID: "dac4d79e-ebfe-453a-8a11-cd3082264301"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 00:07:32.726811 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.726803    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-token" (OuterVolumeSpecName: "istio-token") pod "dac4d79e-ebfe-453a-8a11-cd3082264301" (UID: "dac4d79e-ebfe-453a-8a11-cd3082264301"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 00:07:32.797914 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.797879    2580 generic.go:356] "Generic (PLEG): container finished" podID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerID="1b14cce6276cf092657f9b26dd1ae6b6e031560c54f1ea4ad2dec73737f45db7" exitCode=0
Jul 10 00:07:32.798083 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.797940    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"
Jul 10 00:07:32.798083 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.797956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" event={"ID":"dac4d79e-ebfe-453a-8a11-cd3082264301","Type":"ContainerDied","Data":"1b14cce6276cf092657f9b26dd1ae6b6e031560c54f1ea4ad2dec73737f45db7"}
Jul 10 00:07:32.798083 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.797998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx" event={"ID":"dac4d79e-ebfe-453a-8a11-cd3082264301","Type":"ContainerDied","Data":"90c77397661413555826d27876877415b3a3d5857e63e68102a5448337590c4e"}
Jul 10 00:07:32.798083 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.798020    2580 scope.go:117] "RemoveContainer" containerID="1b14cce6276cf092657f9b26dd1ae6b6e031560c54f1ea4ad2dec73737f45db7"
Jul 10 00:07:32.807415 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.807397    2580 scope.go:117] "RemoveContainer" containerID="9a8a3bda5368d68beb3bf56b6f3c664e5d9dcf181e3f118708a4d0468d22e1a0"
Jul 10 00:07:32.814937 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.814917    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-849b8469f6-m5d4c_b80460ea-9c86-43c1-a59e-97c841313545/manager/0.log"
Jul 10 00:07:32.815499 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.815484    2580 scope.go:117] "RemoveContainer" containerID="1b14cce6276cf092657f9b26dd1ae6b6e031560c54f1ea4ad2dec73737f45db7"
Jul 10 00:07:32.815779 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:07:32.815757    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b14cce6276cf092657f9b26dd1ae6b6e031560c54f1ea4ad2dec73737f45db7\": container with ID starting with 1b14cce6276cf092657f9b26dd1ae6b6e031560c54f1ea4ad2dec73737f45db7 not found: ID does not exist" containerID="1b14cce6276cf092657f9b26dd1ae6b6e031560c54f1ea4ad2dec73737f45db7"
Jul 10 00:07:32.815837 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.815787    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b14cce6276cf092657f9b26dd1ae6b6e031560c54f1ea4ad2dec73737f45db7"} err="failed to get container status \"1b14cce6276cf092657f9b26dd1ae6b6e031560c54f1ea4ad2dec73737f45db7\": rpc error: code = NotFound desc = could not find container \"1b14cce6276cf092657f9b26dd1ae6b6e031560c54f1ea4ad2dec73737f45db7\": container with ID starting with 1b14cce6276cf092657f9b26dd1ae6b6e031560c54f1ea4ad2dec73737f45db7 not found: ID does not exist"
Jul 10 00:07:32.815837 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.815805    2580 scope.go:117] "RemoveContainer" containerID="9a8a3bda5368d68beb3bf56b6f3c664e5d9dcf181e3f118708a4d0468d22e1a0"
Jul 10 00:07:32.816025 ip-10-0-140-8 kubenswrapper[2580]: E0710 00:07:32.816011    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a8a3bda5368d68beb3bf56b6f3c664e5d9dcf181e3f118708a4d0468d22e1a0\": container with ID starting with 9a8a3bda5368d68beb3bf56b6f3c664e5d9dcf181e3f118708a4d0468d22e1a0 not found: ID does not exist" containerID="9a8a3bda5368d68beb3bf56b6f3c664e5d9dcf181e3f118708a4d0468d22e1a0"
Jul 10 00:07:32.816065 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.816030    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8a3bda5368d68beb3bf56b6f3c664e5d9dcf181e3f118708a4d0468d22e1a0"} err="failed to get container status \"9a8a3bda5368d68beb3bf56b6f3c664e5d9dcf181e3f118708a4d0468d22e1a0\": rpc error: code = NotFound desc = could not find container \"9a8a3bda5368d68beb3bf56b6f3c664e5d9dcf181e3f118708a4d0468d22e1a0\": container with ID starting with 9a8a3bda5368d68beb3bf56b6f3c664e5d9dcf181e3f118708a4d0468d22e1a0 not found: ID does not exist"
Jul 10 00:07:32.819430 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.819408    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"]
Jul 10 00:07:32.824670 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.824633    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b0ecf1-openshift-default-575599bf55-dgfgx"]
Jul 10 00:07:32.825812 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.825791    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-podinfo\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:32.825812 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.825812    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-data\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:32.825936 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.825822    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-envoy\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:32.825936 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.825830    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac4d79e-ebfe-453a-8a11-cd3082264301-istio-token\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:32.825936 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.825837    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac4d79e-ebfe-453a-8a11-cd3082264301-workload-certs\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:32.825936 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.825845    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm5q5\" (UniqueName: \"kubernetes.io/projected/dac4d79e-ebfe-453a-8a11-cd3082264301-kube-api-access-qm5q5\") on node \"ip-10-0-140-8.ec2.internal\" DevicePath \"\""
Jul 10 00:07:32.921940 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:32.921866    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6hk6p_6065bde9-3c70-46c7-8c49-f283ee15ecb0/server/0.log"
Jul 10 00:07:33.033698 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:33.033644    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8hbx9_a69d3267-69d9-47eb-b954-e0286f45fe6a/manager/0.log"
Jul 10 00:07:33.263631 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:33.263602    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c55df6879-nq8k8_837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e/copy-manifests/0.log"
Jul 10 00:07:33.269908 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:33.269887    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c55df6879-nq8k8_837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e/manager/0.log"
Jul 10 00:07:33.729327 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:33.729242    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f64fcbf8c-4x9q2_c59cb762-a576-4ddb-b0c8-cf44efba25b1/maas-api/0.log"
Jul 10 00:07:33.839544 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:33.839511    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-fm4kj_0dcc5167-3cad-4948-9e3e-50ebdb78ee5c/postgres/0.log"
Jul 10 00:07:34.418698 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:34.418645    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" path="/var/lib/kubelet/pods/dac4d79e-ebfe-453a-8a11-cd3082264301/volumes"
Jul 10 00:07:34.596327 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:34.596300    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw_24ad7126-c53c-4cc6-bcde-dc0692fe9fe4/extract/0.log"
Jul 10 00:07:34.601395 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:34.601375    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw_24ad7126-c53c-4cc6-bcde-dc0692fe9fe4/util/0.log"
Jul 10 00:07:34.606663 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:34.606631    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw_24ad7126-c53c-4cc6-bcde-dc0692fe9fe4/pull/0.log"
Jul 10 00:07:34.714814 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:34.714733    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw_b501a26c-47ca-4d86-9d34-37c70645e5c5/util/0.log"
Jul 10 00:07:34.720305 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:34.720283    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw_b501a26c-47ca-4d86-9d34-37c70645e5c5/pull/0.log"
Jul 10 00:07:34.725487 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:34.725468    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw_b501a26c-47ca-4d86-9d34-37c70645e5c5/extract/0.log"
Jul 10 00:07:34.831947 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:34.831919    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9_710880cd-b71d-4525-8dc9-d51ba1b001fe/util/0.log"
Jul 10 00:07:34.838094 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:34.838072    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9_710880cd-b71d-4525-8dc9-d51ba1b001fe/pull/0.log"
Jul 10 00:07:34.843468 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:34.843447    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9_710880cd-b71d-4525-8dc9-d51ba1b001fe/extract/0.log"
Jul 10 00:07:34.944157 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:34.944125    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm_95bf198f-c90a-400e-ac71-52ec87b341c3/pull/0.log"
Jul 10 00:07:34.949990 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:34.949965    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm_95bf198f-c90a-400e-ac71-52ec87b341c3/extract/0.log"
Jul 10 00:07:34.955269 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:34.955242    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm_95bf198f-c90a-400e-ac71-52ec87b341c3/util/0.log"
Jul 10 00:07:35.066141 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:35.066109    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b55548785-pmhm9_85f4e702-49a2-42d8-a08f-63f9242654c2/authorino/0.log"
Jul 10 00:07:35.179559 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:35.179530    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5xkp9_981b54dd-b9f2-47e7-aa22-ae0cae9f38ff/manager/0.log"
Jul 10 00:07:35.504287 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:35.504200    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tf7wp_f7f87602-3ecb-463c-9956-2af87f8fffb0/registry-server/0.log"
Jul 10 00:07:35.621923 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:35.621885    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k6zgs_363555ca-801c-4d8f-b5f0-4600041c5a2a/manager/0.log"
Jul 10 00:07:35.840491 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:35.840459    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gmhq7_7e3adeac-a2c0-48be-a349-6e70408d8058/manager/0.log"
Jul 10 00:07:36.392561 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:36.392526    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dff7df579-8sc5p_bf33e8b0-4ed1-4c59-ba89-0493fa839f33/kube-auth-proxy/0.log"
Jul 10 00:07:36.843776 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:36.843751    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-6kzrr_fdd99b0f-9531-4af7-9e2e-32009c97048c/payload-processing/0.log"
Jul 10 00:07:41.401106 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401069    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9scwp/must-gather-hwxst"]
Jul 10 00:07:41.401581 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401441    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy"
Jul 10 00:07:41.401581 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401454    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy"
Jul 10 00:07:41.401581 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401465    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2d9227-a0e7-4f87-b5a9-d045c4b793ff" containerName="test-curl-7224-140222186908528"
Jul 10 00:07:41.401581 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401471    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2d9227-a0e7-4f87-b5a9-d045c4b793ff" containerName="test-curl-7224-140222186908528"
Jul 10 00:07:41.401581 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401479    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy"
Jul 10 00:07:41.401581 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401484    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy"
Jul 10 00:07:41.401581 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401500    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy"
Jul 10 00:07:41.401581 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401510    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy"
Jul 10 00:07:41.402009 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401603    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy"
Jul 10 00:07:41.402009 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401617    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy"
Jul 10 00:07:41.402009 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401628    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e2d9227-a0e7-4f87-b5a9-d045c4b793ff" containerName="test-curl-7224-140222186908528"
Jul 10 00:07:41.402009 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401640    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy"
Jul 10 00:07:41.402009 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401666    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac4d79e-ebfe-453a-8a11-cd3082264301" containerName="istio-proxy"
Jul 10 00:07:41.402009 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401788    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy"
Jul 10 00:07:41.402009 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.401801    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc569f4-ee41-41ee-a4ea-9d4e06cb6688" containerName="istio-proxy"
Jul 10 00:07:41.405004 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.404980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9scwp/must-gather-hwxst"
Jul 10 00:07:41.407503 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.407477    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9scwp\"/\"kube-root-ca.crt\""
Jul 10 00:07:41.407503 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.407484    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9scwp\"/\"openshift-service-ca.crt\""
Jul 10 00:07:41.407700 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.407546    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9scwp\"/\"default-dockercfg-vnv75\""
Jul 10 00:07:41.412765 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.412644    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9scwp/must-gather-hwxst"]
Jul 10 00:07:41.495837 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.495791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/77d3223d-fd65-4750-a710-a5f1b1fd25bd-must-gather-output\") pod \"must-gather-hwxst\" (UID: \"77d3223d-fd65-4750-a710-a5f1b1fd25bd\") " pod="openshift-must-gather-9scwp/must-gather-hwxst"
Jul 10 00:07:41.496065 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.495996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hhlr\" (UniqueName: \"kubernetes.io/projected/77d3223d-fd65-4750-a710-a5f1b1fd25bd-kube-api-access-6hhlr\") pod \"must-gather-hwxst\" (UID: \"77d3223d-fd65-4750-a710-a5f1b1fd25bd\") " pod="openshift-must-gather-9scwp/must-gather-hwxst"
Jul 10 00:07:41.597217 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.597188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/77d3223d-fd65-4750-a710-a5f1b1fd25bd-must-gather-output\") pod \"must-gather-hwxst\" (UID: \"77d3223d-fd65-4750-a710-a5f1b1fd25bd\") " pod="openshift-must-gather-9scwp/must-gather-hwxst"
Jul 10 00:07:41.597386 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.597260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hhlr\" (UniqueName: \"kubernetes.io/projected/77d3223d-fd65-4750-a710-a5f1b1fd25bd-kube-api-access-6hhlr\") pod \"must-gather-hwxst\" (UID: \"77d3223d-fd65-4750-a710-a5f1b1fd25bd\") " pod="openshift-must-gather-9scwp/must-gather-hwxst"
Jul 10 00:07:41.597532 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.597509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/77d3223d-fd65-4750-a710-a5f1b1fd25bd-must-gather-output\") pod \"must-gather-hwxst\" (UID: \"77d3223d-fd65-4750-a710-a5f1b1fd25bd\") " pod="openshift-must-gather-9scwp/must-gather-hwxst"
Jul 10 00:07:41.605137 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.605109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hhlr\" (UniqueName: \"kubernetes.io/projected/77d3223d-fd65-4750-a710-a5f1b1fd25bd-kube-api-access-6hhlr\") pod \"must-gather-hwxst\" (UID: \"77d3223d-fd65-4750-a710-a5f1b1fd25bd\") " pod="openshift-must-gather-9scwp/must-gather-hwxst"
Jul 10 00:07:41.715392 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.715299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9scwp/must-gather-hwxst"
Jul 10 00:07:41.850365 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:41.850335    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9scwp/must-gather-hwxst"]
Jul 10 00:07:41.853013 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:07:41.852989    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77d3223d_fd65_4750_a710_a5f1b1fd25bd.slice/crio-2a1fcccb106463002a926dfdee302d78ea080974e5ebbd84f25a7f755d4ec617 WatchSource:0}: Error finding container 2a1fcccb106463002a926dfdee302d78ea080974e5ebbd84f25a7f755d4ec617: Status 404 returned error can't find the container with id 2a1fcccb106463002a926dfdee302d78ea080974e5ebbd84f25a7f755d4ec617
Jul 10 00:07:42.864798 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:42.864624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9scwp/must-gather-hwxst" event={"ID":"77d3223d-fd65-4750-a710-a5f1b1fd25bd","Type":"ContainerStarted","Data":"8c4f6c33f97cff42f214b81414c3b13df00fcf4f7356e6833bd58148423e6304"}
Jul 10 00:07:42.864798 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:42.864696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9scwp/must-gather-hwxst" event={"ID":"77d3223d-fd65-4750-a710-a5f1b1fd25bd","Type":"ContainerStarted","Data":"5b47f94ee1e9f76555b8991bc136873e0179f451a4ad79204a2c0782ccbc644b"}
Jul 10 00:07:42.864798 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:42.864713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9scwp/must-gather-hwxst" event={"ID":"77d3223d-fd65-4750-a710-a5f1b1fd25bd","Type":"ContainerStarted","Data":"2a1fcccb106463002a926dfdee302d78ea080974e5ebbd84f25a7f755d4ec617"}
Jul 10 00:07:42.880724 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:42.880629    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9scwp/must-gather-hwxst" podStartSLOduration=1.204581951 podStartE2EDuration="1.880609125s" totalImagesPullingTime="676.027174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:07:42.878298509 +0000 UTC m=+2381.022229296" watchObservedRunningTime="2026-07-10 00:07:42.880609125 +0000 UTC m=+2381.024539912"
Jul 10 00:07:44.085701 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:44.085645    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tmmgg_fe6a62a4-df2a-4d08-b134-213d033b0e28/global-pull-secret-syncer/0.log"
Jul 10 00:07:44.165518 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:44.165481    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c7j2c_874f202e-075e-49fd-b6c9-2729f95eb09e/konnectivity-agent/0.log"
Jul 10 00:07:44.233953 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:44.233921    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-8.ec2.internal_3203ef60a382e0f7b0ec847ea9d1892c/haproxy/0.log"
Jul 10 00:07:48.819086 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:48.819041    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw_24ad7126-c53c-4cc6-bcde-dc0692fe9fe4/extract/0.log"
Jul 10 00:07:48.840264 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:48.840234    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw_24ad7126-c53c-4cc6-bcde-dc0692fe9fe4/util/0.log"
Jul 10 00:07:48.865368 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:48.865290    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmfxw_24ad7126-c53c-4cc6-bcde-dc0692fe9fe4/pull/0.log"
Jul 10 00:07:48.922183 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:48.922148    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw_b501a26c-47ca-4d86-9d34-37c70645e5c5/extract/0.log"
Jul 10 00:07:48.946775 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:48.946748    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw_b501a26c-47ca-4d86-9d34-37c70645e5c5/util/0.log"
Jul 10 00:07:48.974722 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:48.974687    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04k4zw_b501a26c-47ca-4d86-9d34-37c70645e5c5/pull/0.log"
Jul 10 00:07:49.005620 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:49.005594    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9_710880cd-b71d-4525-8dc9-d51ba1b001fe/extract/0.log"
Jul 10 00:07:49.029123 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:49.029079    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9_710880cd-b71d-4525-8dc9-d51ba1b001fe/util/0.log"
Jul 10 00:07:49.052188 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:49.050370    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7384mr9_710880cd-b71d-4525-8dc9-d51ba1b001fe/pull/0.log"
Jul 10 00:07:49.095304 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:49.095115    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm_95bf198f-c90a-400e-ac71-52ec87b341c3/extract/0.log"
Jul 10 00:07:49.126008 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:49.125957    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm_95bf198f-c90a-400e-ac71-52ec87b341c3/util/0.log"
Jul 10 00:07:49.147281 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:49.147256    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fjqbm_95bf198f-c90a-400e-ac71-52ec87b341c3/pull/0.log"
Jul 10 00:07:49.578734 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:49.578625    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b55548785-pmhm9_85f4e702-49a2-42d8-a08f-63f9242654c2/authorino/0.log"
Jul 10 00:07:49.610774 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:49.610034    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5xkp9_981b54dd-b9f2-47e7-aa22-ae0cae9f38ff/manager/0.log"
Jul 10 00:07:49.710927 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:49.710892    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tf7wp_f7f87602-3ecb-463c-9956-2af87f8fffb0/registry-server/0.log"
Jul 10 00:07:49.839583 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:49.839475    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k6zgs_363555ca-801c-4d8f-b5f0-4600041c5a2a/manager/0.log"
Jul 10 00:07:49.953308 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:49.952832    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gmhq7_7e3adeac-a2c0-48be-a349-6e70408d8058/manager/0.log"
Jul 10 00:07:51.569438 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:51.569397    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-tr59d_e84df855-0915-471f-8879-b7e0614a0d59/monitoring-plugin/0.log"
Jul 10 00:07:51.771914 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:51.771876    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rq6t6_cbec990d-76e2-4c8e-8955-24270452b2d3/node-exporter/0.log"
Jul 10 00:07:51.801860 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:51.801830    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rq6t6_cbec990d-76e2-4c8e-8955-24270452b2d3/kube-rbac-proxy/0.log"
Jul 10 00:07:51.827710 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:51.827206    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rq6t6_cbec990d-76e2-4c8e-8955-24270452b2d3/init-textfile/0.log"
Jul 10 00:07:52.266205 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:52.266151    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7487767f58-9jstb_e319659e-5d27-4ec3-8a9e-05b77d06b003/thanos-query/0.log"
Jul 10 00:07:52.287442 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:52.287399    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7487767f58-9jstb_e319659e-5d27-4ec3-8a9e-05b77d06b003/kube-rbac-proxy-web/0.log"
Jul 10 00:07:52.307549 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:52.307502    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7487767f58-9jstb_e319659e-5d27-4ec3-8a9e-05b77d06b003/kube-rbac-proxy/0.log"
Jul 10 00:07:52.329827 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:52.329618    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7487767f58-9jstb_e319659e-5d27-4ec3-8a9e-05b77d06b003/prom-label-proxy/0.log"
Jul 10 00:07:52.349912 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:52.349879    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7487767f58-9jstb_e319659e-5d27-4ec3-8a9e-05b77d06b003/kube-rbac-proxy-rules/0.log"
Jul 10 00:07:52.372306 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:52.372269    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7487767f58-9jstb_e319659e-5d27-4ec3-8a9e-05b77d06b003/kube-rbac-proxy-metrics/0.log"
Jul 10 00:07:54.041942 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.041912    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"]
Jul 10 00:07:54.047844 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.047810    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.054972 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.052646    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"]
Jul 10 00:07:54.135103 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.135064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz2zz\" (UniqueName: \"kubernetes.io/projected/74923b41-95e5-46f7-8a28-2dab9fc9c533-kube-api-access-qz2zz\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.135432 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.135406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/74923b41-95e5-46f7-8a28-2dab9fc9c533-podres\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.135564 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.135476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/74923b41-95e5-46f7-8a28-2dab9fc9c533-proc\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.135615 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.135557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74923b41-95e5-46f7-8a28-2dab9fc9c533-sys\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.135615 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.135587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74923b41-95e5-46f7-8a28-2dab9fc9c533-lib-modules\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.236204 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.236161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74923b41-95e5-46f7-8a28-2dab9fc9c533-sys\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.236459 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.236427    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74923b41-95e5-46f7-8a28-2dab9fc9c533-sys\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.236581 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.236560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74923b41-95e5-46f7-8a28-2dab9fc9c533-lib-modules\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.236765 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.236746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz2zz\" (UniqueName: \"kubernetes.io/projected/74923b41-95e5-46f7-8a28-2dab9fc9c533-kube-api-access-qz2zz\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.236923 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.236906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/74923b41-95e5-46f7-8a28-2dab9fc9c533-podres\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.237081 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.237063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/74923b41-95e5-46f7-8a28-2dab9fc9c533-proc\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.237277 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.237260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/74923b41-95e5-46f7-8a28-2dab9fc9c533-proc\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.237380 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.236676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74923b41-95e5-46f7-8a28-2dab9fc9c533-lib-modules\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.237598 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.237573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/74923b41-95e5-46f7-8a28-2dab9fc9c533-podres\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.246069 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.246042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz2zz\" (UniqueName: \"kubernetes.io/projected/74923b41-95e5-46f7-8a28-2dab9fc9c533-kube-api-access-qz2zz\") pod \"perf-node-gather-daemonset-7zh6n\" (UID: \"74923b41-95e5-46f7-8a28-2dab9fc9c533\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.370632 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.370545    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.561311 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.561279    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"]
Jul 10 00:07:54.563623 ip-10-0-140-8 kubenswrapper[2580]: W0710 00:07:54.563593    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod74923b41_95e5_46f7_8a28_2dab9fc9c533.slice/crio-51ba4e4eccc1c9b3690d5b549861ee96ec6c4c37834660c67ef80ad52591b9e7 WatchSource:0}: Error finding container 51ba4e4eccc1c9b3690d5b549861ee96ec6c4c37834660c67ef80ad52591b9e7: Status 404 returned error can't find the container with id 51ba4e4eccc1c9b3690d5b549861ee96ec6c4c37834660c67ef80ad52591b9e7
Jul 10 00:07:54.805358 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.805298    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-4zx77_971ba3dc-711d-4935-bccb-a6c465555b1c/download-server/0.log"
Jul 10 00:07:54.953531 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.953438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n" event={"ID":"74923b41-95e5-46f7-8a28-2dab9fc9c533","Type":"ContainerStarted","Data":"5166e9af8c61df71c543468aa5d41957c401b4a2bf6407a39c45631ea14b451e"}
Jul 10 00:07:54.953531 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.953506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n" event={"ID":"74923b41-95e5-46f7-8a28-2dab9fc9c533","Type":"ContainerStarted","Data":"51ba4e4eccc1c9b3690d5b549861ee96ec6c4c37834660c67ef80ad52591b9e7"}
Jul 10 00:07:54.953800 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.953554    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:07:54.968263 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:54.968189    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n" podStartSLOduration=0.968170089 podStartE2EDuration="968.170089ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:07:54.967259455 +0000 UTC m=+2393.111190220" watchObservedRunningTime="2026-07-10 00:07:54.968170089 +0000 UTC m=+2393.112100876"
Jul 10 00:07:56.488314 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:56.488286    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xll26_b9160d80-c17d-491e-b077-5917e5d6be05/dns/0.log"
Jul 10 00:07:56.507351 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:56.507319    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xll26_b9160d80-c17d-491e-b077-5917e5d6be05/kube-rbac-proxy/0.log"
Jul 10 00:07:56.529862 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:56.529830    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2lhxc_53ac79d2-b750-4cdd-a5bb-dffe1b5c77cb/dns-node-resolver/0.log"
Jul 10 00:07:57.108401 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:57.108372    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4tjx2_20b8cf45-3b5c-46f1-adb0-ebebf097e6e6/node-ca/0.log"
Jul 10 00:07:58.161553 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:58.161518    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dff7df579-8sc5p_bf33e8b0-4ed1-4c59-ba89-0493fa839f33/kube-auth-proxy/0.log"
Jul 10 00:07:58.303857 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:58.303819    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-6kzrr_fdd99b0f-9531-4af7-9e2e-32009c97048c/payload-processing/0.log"
Jul 10 00:07:59.000871 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:59.000828    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c6ztd_15f14655-5e80-48b0-a2d4-81a4ddadaec6/serve-healthcheck-canary/0.log"
Jul 10 00:07:59.486705 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:59.486679    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7b2ch_c4ab2689-0ddb-46e2-95b9-93710ee87dc4/kube-rbac-proxy/0.log"
Jul 10 00:07:59.508331 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:59.508299    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7b2ch_c4ab2689-0ddb-46e2-95b9-93710ee87dc4/exporter/0.log"
Jul 10 00:07:59.530166 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:07:59.530136    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7b2ch_c4ab2689-0ddb-46e2-95b9-93710ee87dc4/extractor/0.log"
Jul 10 00:08:00.972963 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:00.972933    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-7zh6n"
Jul 10 00:08:01.745287 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:01.745259    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-l9vqc_686df5b5-af27-4125-8c70-5b8f0a52afdf/manager/1.log"
Jul 10 00:08:01.746017 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:01.745999    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-l9vqc_686df5b5-af27-4125-8c70-5b8f0a52afdf/manager/2.log"
Jul 10 00:08:01.903027 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:01.902993    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-nmfvk_0719fe1b-21a5-4adc-b802-cd591b01cc17/manager/0.log"
Jul 10 00:08:02.021597 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:02.021566    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-849b8469f6-m5d4c_b80460ea-9c86-43c1-a59e-97c841313545/manager/0.log"
Jul 10 00:08:02.041001 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:02.040977    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6hk6p_6065bde9-3c70-46c7-8c49-f283ee15ecb0/server/0.log"
Jul 10 00:08:02.120495 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:02.120460    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8hbx9_a69d3267-69d9-47eb-b954-e0286f45fe6a/manager/0.log"
Jul 10 00:08:02.192088 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:02.192052    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c55df6879-nq8k8_837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e/manager/0.log"
Jul 10 00:08:02.211685 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:02.211642    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c55df6879-nq8k8_837e94d9-3c7e-473d-ad3d-86b2bf4d8e8e/copy-manifests/0.log"
Jul 10 00:08:03.737985 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:03.737958    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jtrcq_a7d8f739-da68-4dcc-a17e-47ca4a85f8fe/openshift-lws-operator/0.log"
Jul 10 00:08:05.298930 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:05.298898    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dff7df579-8sc5p_bf33e8b0-4ed1-4c59-ba89-0493fa839f33/kube-auth-proxy/0.log"
Jul 10 00:08:05.421430 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:05.421394    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-6kzrr_fdd99b0f-9531-4af7-9e2e-32009c97048c/payload-processing/0.log"
Jul 10 00:08:06.802740 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:06.802708    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dff7df579-8sc5p_bf33e8b0-4ed1-4c59-ba89-0493fa839f33/kube-auth-proxy/0.log"
Jul 10 00:08:06.918892 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:06.918857    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-6kzrr_fdd99b0f-9531-4af7-9e2e-32009c97048c/payload-processing/0.log"
Jul 10 00:08:10.452629 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:10.452598    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p96p7_8083d0cd-a5a4-42fe-863a-e70e5b0e8a55/kube-multus-additional-cni-plugins/0.log"
Jul 10 00:08:10.470883 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:10.470857    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p96p7_8083d0cd-a5a4-42fe-863a-e70e5b0e8a55/egress-router-binary-copy/0.log"
Jul 10 00:08:10.490663 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:10.490633    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p96p7_8083d0cd-a5a4-42fe-863a-e70e5b0e8a55/cni-plugins/0.log"
Jul 10 00:08:10.509011 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:10.508980    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p96p7_8083d0cd-a5a4-42fe-863a-e70e5b0e8a55/bond-cni-plugin/0.log"
Jul 10 00:08:10.528016 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:10.527982    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p96p7_8083d0cd-a5a4-42fe-863a-e70e5b0e8a55/routeoverride-cni/0.log"
Jul 10 00:08:10.546998 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:10.546975    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p96p7_8083d0cd-a5a4-42fe-863a-e70e5b0e8a55/whereabouts-cni-bincopy/0.log"
Jul 10 00:08:10.565563 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:10.565528    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p96p7_8083d0cd-a5a4-42fe-863a-e70e5b0e8a55/whereabouts-cni/0.log"
Jul 10 00:08:10.606275 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:10.606245    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-grfjv_76444aa9-03ec-4980-97f4-dec9f17bbdc6/kube-multus/0.log"
Jul 10 00:08:10.730567 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:10.730485    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m5bj7_1e7add64-a9cc-4acc-9102-a8c5efbbc5a8/network-metrics-daemon/0.log"
Jul 10 00:08:10.749445 ip-10-0-140-8 kubenswrapper[2580]: I0710 00:08:10.749416    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m5bj7_1e7add64-a9cc-4acc-9102-a8c5efbbc5a8/kube-rbac-proxy/0.log"