Jul 16 19:58:21.895043 ip-10-0-128-109 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 16 19:58:21.895060 ip-10-0-128-109 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 16 19:58:21.895069 ip-10-0-128-109 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 16 19:58:21.895405 ip-10-0-128-109 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 16 19:58:31.467955 ip-10-0-128-109 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Jul 16 19:58:31.467970 ip-10-0-128-109 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 05c311206bc640d9b7f914bf743552a3 --
Jul 16 20:00:21.639011 ip-10-0-128-109 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 20:00:22.011023 ip-10-0-128-109 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 20:00:22.011023 ip-10-0-128-109 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 20:00:22.011023 ip-10-0-128-109 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 20:00:22.011023 ip-10-0-128-109 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 20:00:22.011023 ip-10-0-128-109 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 20:00:22.011759 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.011083    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 20:00:22.014443 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014427    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:00:22.014443 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014443    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014447    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014450    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014453    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014455    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014458    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014461    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014463    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014466    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014469    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014471    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014474    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014479    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014482    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014485    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014488    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014491    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014493    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014496    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:00:22.014503 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014498    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014501    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014504    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014507    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014510    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014513    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014516    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014518    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014521    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014524    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014527    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014529    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014532    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014535    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014537    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014540    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014542    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014544    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014547    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014551    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:00:22.014988 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014554    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014557    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014561    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014563    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014566    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014568    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014571    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014573    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014576    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014578    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014581    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014583    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014587    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014590    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014592    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014596    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014598    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014602    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014605    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014607    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:00:22.015455 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014610    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014612    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014615    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014617    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014620    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014622    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014625    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014627    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014630    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014632    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014635    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014638    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014641    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014644    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014646    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014648    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014651    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014653    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014656    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:00:22.015948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014659    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014662    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014665    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014670    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014673    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014676    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014679    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014681    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014684    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014687    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014690    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014692    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014710    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014715    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014720    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014723    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014725    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014728    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014730    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.014733    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:00:22.016433 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015452    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015460    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015464    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015467    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015470    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015474    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015477    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015479    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015482    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015484    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015487    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015490    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015492    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015495    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015497    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015500    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015502    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015505    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015508    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015510    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:00:22.017052 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015513    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015515    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015518    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015520    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015523    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015525    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015528    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015530    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015533    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015535    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015538    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015540    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015543    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015545    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015548    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015551    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015553    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015556    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015559    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:00:22.017539 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015561    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015564    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015566    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015569    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015571    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015574    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015577    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015579    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015582    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015584    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015588    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015590    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015593    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015595    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015598    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015600    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015603    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015605    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015609    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:00:22.018019 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015614    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015616    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015619    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015622    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015624    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015628    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015632    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015634    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015637    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015640    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015643    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015645    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015648    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015651    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015654    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015657    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015659    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015662    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015664    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:00:22.018473 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015667    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015669    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015672    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015674    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015677    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015679    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015682    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015684    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015688    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015690    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015693    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015716    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015721    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015725    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015730    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015733    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015736    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015739    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015741    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015744    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:00:22.018969 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015750    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.015753    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016128    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016133    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016135    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016138    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016141    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016144    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016146    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016149    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016151    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016154    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016156    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016159    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016161    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016165    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016168    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016170    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016173    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016175    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:00:22.019440 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016178    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016181    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016183    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016186    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016188    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016191    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016193    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016196    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016198    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016201    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016203    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016206    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016210    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016212    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016215    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016217    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016220    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016222    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016225    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016235    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:00:22.019946 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016238    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016240    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016243    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016245    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016248    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016250    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016253    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016256    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016258    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016261    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016263    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016266    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016268    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016271    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016273    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016276    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016278    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016281    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016283    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016286    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:00:22.020417 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016288    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016290    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016294    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016296    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016302    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016306    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016310    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016314    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016317    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016320    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016323    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016326    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016333    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016336    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016339    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016341    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016344    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016346    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016349    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016353    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:00:22.020982 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016361    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016363    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016367    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016370    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016374    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016377    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016380    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016383    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016386    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016389    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016391    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016394    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016396    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016399    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016401    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016404    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016408    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016411    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016413    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:00:22.021466 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016416    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.016418    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016487    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016493    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016524    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016529    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016544    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016547    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016551    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016555    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016558    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016561    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016565    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016568    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016571    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016574    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016577    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016580    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016582    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016585    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016592    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016595    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016598    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016601    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 20:00:22.021948 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016604    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016607    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016610    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016613    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016616    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016620    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016623    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016626    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016629    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016632    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016636    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016638    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016641    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016644    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016647    2576 flags.go:64] FLAG: --enable-server="true"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016658    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016664    2576 flags.go:64] FLAG: --event-burst="100"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016667    2576 flags.go:64] FLAG: --event-qps="50"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016671    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016674    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016677    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016680    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016683    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016686    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016689    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 16 20:00:22.022502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016691    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016708    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016712    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016715    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016717    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016720    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016723    2576 flags.go:64] FLAG: --feature-gates=""
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016726    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016729    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016732    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016735    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016739    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016743    2576 flags.go:64] FLAG: --help="false"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016746    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016749    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016751    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016754    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016757    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016761    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016764    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016766    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016769    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016772    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 20:00:22.023133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016782    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016785    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016788    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016790    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016793    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016796    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016799    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016802    2576 flags.go:64] FLAG: --lock-file=""
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016804    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016807    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016810    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016815    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016818    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016820    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016823    2576 flags.go:64] FLAG: --logging-format="text"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016826    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016829    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016832    2576 flags.go:64] FLAG: --manifest-url=""
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016834    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016838    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016841    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016846    2576 flags.go:64] FLAG: --max-pods="110"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016849    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016852    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016855    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 20:00:22.023728 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016857    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016860    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016863    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016866    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016872    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016875    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016878    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016881    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016889    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016894    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016896    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016900    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016903    2576 flags.go:64] FLAG: --port="10250"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016906    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016908    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0efe00a4601f947d5"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016911    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016914    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016917    2576 flags.go:64] FLAG: --register-node="true"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016920    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016923    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016926    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016929    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016931    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016935    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016938    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 20:00:22.024334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016941    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016944    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016946    2576 flags.go:64] FLAG: --runonce="false"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016949    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016954    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016957    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016959    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016962    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016965    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016968    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016971    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016973    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016976    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016979    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016982    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016985    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016989    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016992    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.016997    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.017000    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.017003    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.017006    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.017009    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.017012    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.017015    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 20:00:22.025004 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.017017    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.017020    2576 flags.go:64] FLAG: --v="2"
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.017023    2576 flags.go:64] FLAG: --version="false"
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.017027    2576 flags.go:64] FLAG: --vmodule=""
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.017031    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.017034    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017154    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017158    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017161    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017164    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017166    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017170    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017173    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017176    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017180    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017182    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017185    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017189    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017192    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017196    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017199    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:00:22.025606 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017202    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017204    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017207    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017210    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017213    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017216    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017219    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017221    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017224    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017226    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017229    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017231    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017233    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017236    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017239    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017242    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017244    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017247    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017249    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017252    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:00:22.026163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017255    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017257    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017260    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017262    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017265    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017267    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017270    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017273    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017275    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017278    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017280    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017283    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017285    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017288    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017291    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017294    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017297    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017299    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017302    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017304    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:00:22.026637 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017307    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017309    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017311    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017314    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017316    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017319    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017322    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017325    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017327    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017330    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017332    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017334    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017337    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017339    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017342    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017344    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017347    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017350    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017354    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:00:22.027151 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017358    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017361    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017364    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017366    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017369    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017372    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017374    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017378    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017381    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017383    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017386    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017389    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017391    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017394    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017396    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017398    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017401    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017404    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017406    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017409    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:00:22.027602 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017411    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017413    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017416    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017418    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.017421    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.017426    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.024506    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.024520    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.026148    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.026160    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026218    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026224    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026227    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026231    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026234    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:00:22.028095 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026236    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026239    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026242    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026244    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026247    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026250    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026252    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026256    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026258    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026263    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026266    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026269    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026272    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026275    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026278    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026281    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026285    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026287    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026290    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026293    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:00:22.028464 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026296    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026299    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026301    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026304    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026307    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026310    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026313    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026315    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026318    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026320    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026323    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026326    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026328    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026331    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026334    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026337    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026339    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026342    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026344    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026347    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:00:22.028979 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026350    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026353    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026355    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026358    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026360    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026362    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026365    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026367    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026370    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026372    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026375    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026377    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026380    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026382    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026385    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026387    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026391    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026394    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026396    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:00:22.029446 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026399    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026401    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026404    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026407    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026409    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026412    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026414    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026417    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026421    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026424    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026427    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026430    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026432    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026435    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026438    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026441    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026443    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026446    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026448    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026451    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:00:22.030022 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026453    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026456    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026459    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026461    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026464    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026466    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026468    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026471    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026474    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026477    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026479    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026483    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026486    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026489    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:00:22.030497 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026492    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.026497    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026606    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026611    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026614    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026617    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026619    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026623    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026627    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026630    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026633    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026636    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026638    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026642    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026644    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:00:22.030855 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026647    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026650    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026652    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026655    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026657    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026659    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026663    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026667    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026669    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026672    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026675    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026678    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026680    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026683    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026685    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026688    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026690    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026693    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026695    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026714    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:00:22.031231 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026717    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026719    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026723    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026726    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026729    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026732    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026735    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026737    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026740    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026742    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026745    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026748    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026750    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026753    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026755    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026758    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026760    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026762    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026765    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:00:22.031710 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026767    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026770    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026773    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026775    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026778    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026781    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026784    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026786    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026789    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026791    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026794    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026797    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026799    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026802    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026804    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026806    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026809    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026811    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026814    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026817    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:00:22.032166 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026819    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026822    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026824    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026827    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026829    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026832    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026834    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026837    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026839    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026842    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026844    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026847    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026849    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026852    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026854    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026857    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026859    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026862    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026864    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026867    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:00:22.032650 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026869    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:00:22.033186 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026872    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:00:22.033186 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026874    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:00:22.033186 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026884    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:00:22.033186 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026888    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:00:22.033186 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026890    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:00:22.033186 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:22.026893    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:00:22.033186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.026898    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 20:00:22.033186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.026992    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 20:00:22.033186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.028851    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 20:00:22.033186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.030146    2576 server.go:1019] "Starting client certificate rotation"
Jul 16 20:00:22.033186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.030250    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 20:00:22.033186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.030297    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 20:00:22.033186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.032318    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 20:00:22.035298 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.035276    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 20:00:22.047309 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.047288    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 16 20:00:22.049371 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.049355    2576 log.go:25] "Validated CRI v1 image API"
Jul 16 20:00:22.050210 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.050192    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 20:00:22.053825 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.053801    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a38d06c1-dbe3-499e-9d67-a5efef9a2eea:/dev/nvme0n1p4 fa4768e7-a1ce-413e-90b1-6c8e0959d64a:/dev/nvme0n1p3]
Jul 16 20:00:22.053897 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.053824    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 20:00:22.057623 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.057606    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 20:00:22.058886 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.058771    2576 manager.go:216] Machine: {Timestamp:2026-07-16 20:00:22.057434771 +0000 UTC m=+0.308055520 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3087117 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ff8be7fd27b1a8543fd8ebe784d9a SystemUUID:ec2ff8be-7fd2-7b1a-8543-fd8ebe784d9a BootID:05c31120-6bc6-40d9-b7f9-14bf743552a3 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ef:1c:fa:2b:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ef:1c:fa:2b:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:f9:b6:2d:3a:9a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 20:00:22.058886 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.058881    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 20:00:22.059007 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.058957    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 20:00:22.061428 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.061404    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 20:00:22.061593 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.061431    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-109.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 20:00:22.062106 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.062094    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 20:00:22.062159 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.062111    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 20:00:22.062159 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.062122    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 20:00:22.062725 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.062695    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 20:00:22.063972 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.063960    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 20:00:22.067488 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.067474    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 20:00:22.067616 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.067605    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 20:00:22.067671 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.067643    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 20:00:22.067671 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.067658    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 20:00:22.067785 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.067673    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 20:00:22.067785 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.067685    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 16 20:00:22.067785 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.067717    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 20:00:22.068082 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.068068    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 20:00:22.068145 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.068091    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 20:00:22.070293 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.070279    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 20:00:22.071552 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.071536    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 20:00:22.071635 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.071575    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 20:00:22.072134 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072121    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 20:00:22.072197 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072141    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 20:00:22.072197 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072150    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 20:00:22.072197 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072159    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 20:00:22.072197 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072171    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 20:00:22.072197 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072181    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 20:00:22.072387 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072221    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 20:00:22.072387 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072230    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 20:00:22.072387 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072241    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 20:00:22.072387 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072249    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 20:00:22.072387 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072268    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 20:00:22.072387 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072280    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 20:00:22.072954 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072944    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 20:00:22.073003 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.072957    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 20:00:22.076565 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.076548    2576 server.go:1268] "Started kubelet"
Jul 16 20:00:22.076682 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.076634    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 20:00:22.076760 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.076681    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 20:00:22.076802 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.076768    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 20:00:22.077516 ip-10-0-128-109 systemd[1]: Started Kubernetes Kubelet.
Jul 16 20:00:22.078390 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.078318    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 20:00:22.079367 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.079351    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 20:00:22.079467 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.079448    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qwsbb"
Jul 16 20:00:22.079923 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.079903    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-109.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 20:00:22.080011 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.079932    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 20:00:22.080011 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.079932    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 20:00:22.082884 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.082832    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 20:00:22.083890 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.082783    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-109.ec2.internal.18c2dd9220bf5468  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-109.ec2.internal,UID:ip-10-0-128-109.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-109.ec2.internal,},FirstTimestamp:2026-07-16 20:00:22.076511336 +0000 UTC m=+0.327132086,LastTimestamp:2026-07-16 20:00:22.076511336 +0000 UTC m=+0.327132086,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-109.ec2.internal,}"
Jul 16 20:00:22.085383 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.085369    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qwsbb"
Jul 16 20:00:22.085484 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.085472    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 20:00:22.085531 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.085514    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 20:00:22.085685 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.085664    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-109.ec2.internal\" not found"
Jul 16 20:00:22.086037 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.086013    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 20:00:22.086118 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.086047    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 20:00:22.086169 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.086143    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 20:00:22.086169 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.086155    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 20:00:22.086169 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.086147    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 20:00:22.087051 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.086805    2576 factory.go:55] Registering systemd factory
Jul 16 20:00:22.087051 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.086845    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 16 20:00:22.088168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.088150    2576 factory.go:153] Registering CRI-O factory
Jul 16 20:00:22.088168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.088170    2576 factory.go:223] Registration of the crio container factory successfully
Jul 16 20:00:22.088306 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.088221    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 20:00:22.088306 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.088246    2576 factory.go:103] Registering Raw factory
Jul 16 20:00:22.088306 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.088262    2576 manager.go:1196] Started watching for new ooms in manager
Jul 16 20:00:22.088580 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.088564    2576 manager.go:349] Starting recovery of all containers
Jul 16 20:00:22.089310 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.089279    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 20:00:22.089462 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.089441    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 20:00:22.098589 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.098423    2576 manager.go:354] Recovery completed
Jul 16 20:00:22.100189 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.100162    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 20:00:22.104006 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.103989    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:00:22.105970 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.105949    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:00:22.106042 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.105988    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:00:22.106042 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.106005    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:00:22.106596 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.106580    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 20:00:22.106596 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.106596    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 20:00:22.106692 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.106614    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 20:00:22.106692 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.106622    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:00:22.107334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.107320    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:00:22.107389 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.107347    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:00:22.107389 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.107356    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:00:22.108690 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.108677    2576 policy_none.go:49] "None policy: Start"
Jul 16 20:00:22.108770 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.108711    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 20:00:22.108770 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.108725    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 20:00:22.110635 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.110622    2576 policy_none.go:47] "Start"
Jul 16 20:00:22.154343 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.154326    2576 manager.go:337] "Starting Device Plugin manager"
Jul 16 20:00:22.154454 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.154364    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 20:00:22.154454 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.154377    2576 server.go:87] "Starting device plugin registration server"
Jul 16 20:00:22.154637 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.154624    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 20:00:22.154790 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.154637    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 20:00:22.154790 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.154732    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 20:00:22.154913 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.154808    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 20:00:22.154913 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.154818    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 20:00:22.155341 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.155321    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 20:00:22.155401 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.155379    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-109.ec2.internal\" not found"
Jul 16 20:00:22.238181 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.238158    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 20:00:22.240079 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.238199    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 20:00:22.240079 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.238219    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 20:00:22.240079 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.238249    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 20:00:22.240587 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.240569    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:00:22.255592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.255578    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:00:22.256366 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.256350    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:00:22.256441 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.256382    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:00:22.256441 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.256395    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:00:22.256441 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.256418    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.266055 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.266006    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.266055 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.266027    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-109.ec2.internal\": node \"ip-10-0-128-109.ec2.internal\" not found"
Jul 16 20:00:22.284723 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.284691    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-109.ec2.internal\" not found"
Jul 16 20:00:22.339133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.339089    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-109.ec2.internal"]
Jul 16 20:00:22.339247 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.339198    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:00:22.340993 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.340978    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:00:22.341082 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.341010    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:00:22.341082 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.341019    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:00:22.345042 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.345027    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:00:22.345147 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.345133    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.345196 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.345161    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:00:22.345804 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.345787    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:00:22.345888 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.345813    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:00:22.345888 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.345823    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:00:22.345888 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.345788    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:00:22.345992 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.345895    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:00:22.345992 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.345906    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:00:22.347826 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.347809    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.347918 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.347835    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:00:22.348483 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.348466    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:00:22.348559 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.348497    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:00:22.348559 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.348511    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:00:22.363274 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.363251    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-109.ec2.internal\" not found" node="ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.367263 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.367247    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-109.ec2.internal\" not found" node="ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.385558 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.385539    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-109.ec2.internal\" not found"
Jul 16 20:00:22.485826 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.485794    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-109.ec2.internal\" not found"
Jul 16 20:00:22.486980 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.486963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6979e07120a9f54f8d29e294507849f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal\" (UID: \"6979e07120a9f54f8d29e294507849f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.487035 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.486990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6979e07120a9f54f8d29e294507849f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal\" (UID: \"6979e07120a9f54f8d29e294507849f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.487035 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.487008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93b88c9554c3d1c5c57b49c2ee4f8f5e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-109.ec2.internal\" (UID: \"93b88c9554c3d1c5c57b49c2ee4f8f5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.586443 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.586363    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-109.ec2.internal\" not found"
Jul 16 20:00:22.587497 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.587481    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6979e07120a9f54f8d29e294507849f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal\" (UID: \"6979e07120a9f54f8d29e294507849f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.587543 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.587505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6979e07120a9f54f8d29e294507849f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal\" (UID: \"6979e07120a9f54f8d29e294507849f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.587543 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.587521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93b88c9554c3d1c5c57b49c2ee4f8f5e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-109.ec2.internal\" (UID: \"93b88c9554c3d1c5c57b49c2ee4f8f5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.587606 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.587566    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93b88c9554c3d1c5c57b49c2ee4f8f5e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-109.ec2.internal\" (UID: \"93b88c9554c3d1c5c57b49c2ee4f8f5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.588022 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.588005    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6979e07120a9f54f8d29e294507849f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal\" (UID: \"6979e07120a9f54f8d29e294507849f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.588068 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.588054    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6979e07120a9f54f8d29e294507849f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal\" (UID: \"6979e07120a9f54f8d29e294507849f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.665718 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.665659    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.670967 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.670527    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:22.686453 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.686429    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-109.ec2.internal\" not found"
Jul 16 20:00:22.786845 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.786813    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-109.ec2.internal\" not found"
Jul 16 20:00:22.887303 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.887241    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-109.ec2.internal\" not found"
Jul 16 20:00:22.962330 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:22.962302    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:00:22.987665 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:22.987626    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-109.ec2.internal\" not found"
Jul 16 20:00:23.029349 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:23.029317    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 20:00:23.029957 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:23.029467    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 20:00:23.029957 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:23.029526    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 20:00:23.086399 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:23.086364    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:00:23.088226 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:23.088194    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-109.ec2.internal\" not found"
Jul 16 20:00:23.092671 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:23.092635    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 19:55:22 +0000 UTC" deadline="2028-01-27 18:44:06.101785174 +0000 UTC"
Jul 16 20:00:23.092671 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:23.092665    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13438h43m43.009123695s"
Jul 16 20:00:23.107408 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:23.107381    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:00:23.186921 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:23.186831    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:23.201420 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:23.201392    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 20:00:23.202372 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:23.202358    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal"
Jul 16 20:00:23.213503 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:23.213481    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 20:00:23.259758 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:23.259728    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6979e07120a9f54f8d29e294507849f7.slice/crio-da2b3609ff1854ffb76ed3261d7c1276a66eea75a88b11112b62602e0c545bac WatchSource:0}: Error finding container da2b3609ff1854ffb76ed3261d7c1276a66eea75a88b11112b62602e0c545bac: Status 404 returned error can't find the container with id da2b3609ff1854ffb76ed3261d7c1276a66eea75a88b11112b62602e0c545bac
Jul 16 20:00:23.264750 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:23.264735    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 20:00:23.462840 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:23.462802    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b88c9554c3d1c5c57b49c2ee4f8f5e.slice/crio-f2b0c6c542b725450b5abc57be8ecc67f11c67eddf190b4563b11ec6a514e16d WatchSource:0}: Error finding container f2b0c6c542b725450b5abc57be8ecc67f11c67eddf190b4563b11ec6a514e16d: Status 404 returned error can't find the container with id f2b0c6c542b725450b5abc57be8ecc67f11c67eddf190b4563b11ec6a514e16d
Jul 16 20:00:24.012480 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.012448    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:00:24.071105 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.071078    2576 apiserver.go:52] "Watching apiserver"
Jul 16 20:00:24.080290 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.080261    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 20:00:24.080849 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.080818    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-109.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rjqp9","openshift-dns/node-resolver-vqw9q","openshift-image-registry/node-ca-5h765","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal","openshift-multus/multus-additional-cni-plugins-4rwcb","openshift-multus/network-metrics-daemon-6ffx6","openshift-network-diagnostics/network-check-target-7jgvl","kube-system/global-pull-secret-syncer-szlh8","kube-system/konnectivity-agent-m74x5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx","openshift-multus/multus-c5bsz","openshift-network-operator/iptables-alerter-gwntx","openshift-ovn-kubernetes/ovnkube-node-srq56"]
Jul 16 20:00:24.083884 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.083868    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m74x5"
Jul 16 20:00:24.086290 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.086267    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 20:00:24.086522 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.086454    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.086719 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.086696    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 20:00:24.086961 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.086936    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 20:00:24.087073 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.087056    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gc7dg\""
Jul 16 20:00:24.088932 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.088902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.089142 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.089123    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 20:00:24.089495 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.089480    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 20:00:24.089695 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.089678    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tklct\""
Jul 16 20:00:24.089991 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.089970    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 20:00:24.089991 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.089985    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 20:00:24.090188 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.090041    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 20:00:24.090355 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.090337    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 20:00:24.091138 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.091115    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r2jcn\""
Jul 16 20:00:24.091236 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.091196    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 20:00:24.091236 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.091208    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 20:00:24.091803 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.091544    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 20:00:24.091803 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.091550    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5h765"
Jul 16 20:00:24.094150 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.093956    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 20:00:24.094150 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.094069    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.094378 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.094359    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 20:00:24.094480 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.094410    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 20:00:24.094542 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.094528    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wcg4k\""
Jul 16 20:00:24.095902 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.095879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-slash\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.095990 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.095921    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-run-netns\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.095990 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.095941    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2e309c4f-2923-4a09-9b32-94a389560a90-ovnkube-config\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.095990 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.095964    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-etc-selinux\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.095990 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.095987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2e309c4f-2923-4a09-9b32-94a389560a90-env-overrides\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096163 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-sys-fs\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.096163 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwg7d\" (UniqueName: \"kubernetes.io/projected/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-kube-api-access-hwg7d\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.096163 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-run-ovn-kubernetes\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096163 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096078    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096163 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e309c4f-2923-4a09-9b32-94a389560a90-ovn-node-metrics-cert\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096381 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-node-log\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096381 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2e309c4f-2923-4a09-9b32-94a389560a90-ovnkube-script-lib\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096381 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-device-dir\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.096381 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-etc-openvswitch\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096381 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096293    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-run-openvswitch\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096381 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-log-socket\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096687 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.096687 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-var-lib-openvswitch\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096687 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-cni-bin\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096687 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-run-systemd\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096687 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096517    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-run-ovn\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096687 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096541    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5f6f\" (UniqueName: \"kubernetes.io/projected/2e309c4f-2923-4a09-9b32-94a389560a90-kube-api-access-l5f6f\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096687 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096547    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:24.096687 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c-agent-certs\") pod \"konnectivity-agent-m74x5\" (UID: \"e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c\") " pod="kube-system/konnectivity-agent-m74x5"
Jul 16 20:00:24.096687 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-kubelet\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.096687 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.096623    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:24.096687 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-systemd-units\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.097131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096704    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-cni-netd\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.097131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-socket-dir\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.097131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096775    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-registration-dir\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.097131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.096816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c-konnectivity-ca\") pod \"konnectivity-agent-m74x5\" (UID: \"e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c\") " pod="kube-system/konnectivity-agent-m74x5"
Jul 16 20:00:24.097131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.097076    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-58v9s\""
Jul 16 20:00:24.097131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.097088    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 20:00:24.097380 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.097154    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 20:00:24.097460 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.097442    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 20:00:24.097518 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.097478    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 20:00:24.097518 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.097486    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 20:00:24.101862 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.099082    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:24.101862 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.099160    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:24.101862 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.100142    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 20:00:24.103072 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.103048    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:24.103242 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.103219    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:24.108432 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.108414    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vqw9q"
Jul 16 20:00:24.110741 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.110721    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8b2kh\""
Jul 16 20:00:24.111077 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.111060    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 20:00:24.111197 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.111176    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 20:00:24.111335 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.111313    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.113738 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.113719    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 20:00:24.113896 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.113812    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4smtm\""
Jul 16 20:00:24.114013 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.113994    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gwntx"
Jul 16 20:00:24.116396 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.116365    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.116690 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.116664    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 20:00:24.116841 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.116820    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 20:00:24.116989 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.116968    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 20:00:24.117346 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.117316    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v9cdl\""
Jul 16 20:00:24.119320 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.119301    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 20:00:24.119417 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.119331    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pp4gc\""
Jul 16 20:00:24.119582 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.119564    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 20:00:24.124792 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.124772    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kx6tl"
Jul 16 20:00:24.134150 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.134120    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kx6tl"
Jul 16 20:00:24.188230 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.188203    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 20:00:24.197809 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.197779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-var-lib-kubelet\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.197809 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.197816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1df43134-62af-464b-9f40-304412852ec9-host-slash\") pod \"iptables-alerter-gwntx\" (UID: \"1df43134-62af-464b-9f40-304412852ec9\") " pod="openshift-network-operator/iptables-alerter-gwntx"
Jul 16 20:00:24.198027 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.197844    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29d77657-97fd-4e5a-97d9-63dea83fadf9-cnibin\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.198027 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.197911    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwg7d\" (UniqueName: \"kubernetes.io/projected/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-kube-api-access-hwg7d\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.198027 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.197950    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:24.198027 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.197979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-kubernetes\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.198027 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-sysctl-d\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.198299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198035    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-run-ovn-kubernetes\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.198299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198064    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e309c4f-2923-4a09-9b32-94a389560a90-ovn-node-metrics-cert\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.198299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198090    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-modprobe-d\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.198299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198116    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9657416-219b-4d50-a3ff-7db1065072aa-tmp\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.198299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-node-log\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.198299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198160    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-run-ovn-kubernetes\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.198299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-hostroot\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.198299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d77657-97fd-4e5a-97d9-63dea83fadf9-system-cni-dir\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.198299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198237    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29d77657-97fd-4e5a-97d9-63dea83fadf9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.198299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198246    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-node-log\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.198299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198264    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-sysctl-conf\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.198299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198289    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-host\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9657416-219b-4d50-a3ff-7db1065072aa-etc-tuned\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-run-openvswitch\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198440    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-run-openvswitch\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-log-socket\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/29d77657-97fd-4e5a-97d9-63dea83fadf9-os-release\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198525    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpzjn\" (UniqueName: \"kubernetes.io/projected/29d77657-97fd-4e5a-97d9-63dea83fadf9-kube-api-access-cpzjn\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-var-lib-openvswitch\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198627    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-run-k8s-cni-cncf-io\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198645    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-var-lib-cni-multus\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-sysconfig\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198702    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-run-multus-certs\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-systemd\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198793    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-var-lib-openvswitch\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-lib-modules\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.198844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198848    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-log-socket\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-kubelet\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198916    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-kubelet\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-systemd-units\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198942    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-systemd-units\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198969    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-socket-dir\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.198998    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-registration-dir\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-cnibin\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199053    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/29d77657-97fd-4e5a-97d9-63dea83fadf9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199080    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpskv\" (UniqueName: \"kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv\") pod \"network-check-target-7jgvl\" (UID: \"ac9ea87f-36ee-4233-8037-2e3802ae7ae6\") " pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c-konnectivity-ca\") pod \"konnectivity-agent-m74x5\" (UID: \"e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c\") " pod="kube-system/konnectivity-agent-m74x5"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-slash\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-run-netns\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-var-lib-kubelet\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2e309c4f-2923-4a09-9b32-94a389560a90-env-overrides\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199240    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-sys-fs\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-etc-kubernetes\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.199666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-registration-dir\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.200537 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.200537 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199320    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-slash\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.200537 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199844    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-sys-fs\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.200537 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199213    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-socket-dir\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.200537 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.199910    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-run-netns\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.200537 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.200098    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.200537 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.200267    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c-konnectivity-ca\") pod \"konnectivity-agent-m74x5\" (UID: \"e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c\") " pod="kube-system/konnectivity-agent-m74x5"
Jul 16 20:00:24.200537 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.200448    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2e309c4f-2923-4a09-9b32-94a389560a90-env-overrides\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.201067 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-system-cni-dir\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.201208 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs\") pod \"network-metrics-daemon-6ffx6\" (UID: \"0238483d-389c-44cf-92b1-7f8cf040159e\") " pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:24.201208 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201113    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2e309c4f-2923-4a09-9b32-94a389560a90-ovnkube-script-lib\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.201208 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201151    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-device-dir\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.201208 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-cni-binary-copy\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.201468 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-multus-daemon-config\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.201468 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201244    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9711524-15cd-4165-854a-1453f37e4abc-dbus\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:24.201468 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-run\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.201468 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201315    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-etc-openvswitch\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.201468 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.201468 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-multus-socket-dir-parent\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.201468 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-run-netns\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.201468 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1df43134-62af-464b-9f40-304412852ec9-iptables-alerter-script\") pod \"iptables-alerter-gwntx\" (UID: \"1df43134-62af-464b-9f40-304412852ec9\") " pod="openshift-network-operator/iptables-alerter-gwntx"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29d77657-97fd-4e5a-97d9-63dea83fadf9-cni-binary-copy\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201526    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9711524-15cd-4165-854a-1453f37e4abc-kubelet-config\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-cni-bin\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv789\" (UniqueName: \"kubernetes.io/projected/1df43134-62af-464b-9f40-304412852ec9-kube-api-access-tv789\") pod \"iptables-alerter-gwntx\" (UID: \"1df43134-62af-464b-9f40-304412852ec9\") " pod="openshift-network-operator/iptables-alerter-gwntx"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201625    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c98ae9-ea2f-4de5-9b8b-06cc63a945d7-tmp-dir\") pod \"node-resolver-vqw9q\" (UID: \"08c98ae9-ea2f-4de5-9b8b-06cc63a945d7\") " pod="openshift-dns/node-resolver-vqw9q"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vclk\" (UniqueName: \"kubernetes.io/projected/08c98ae9-ea2f-4de5-9b8b-06cc63a945d7-kube-api-access-6vclk\") pod \"node-resolver-vqw9q\" (UID: \"08c98ae9-ea2f-4de5-9b8b-06cc63a945d7\") " pod="openshift-dns/node-resolver-vqw9q"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-sys\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97cc28a3-e813-4407-bad7-8f53dec993f4-host\") pod \"node-ca-5h765\" (UID: \"97cc28a3-e813-4407-bad7-8f53dec993f4\") " pod="openshift-image-registry/node-ca-5h765"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkf46\" (UniqueName: \"kubernetes.io/projected/97cc28a3-e813-4407-bad7-8f53dec993f4-kube-api-access-mkf46\") pod \"node-ca-5h765\" (UID: \"97cc28a3-e813-4407-bad7-8f53dec993f4\") " pod="openshift-image-registry/node-ca-5h765"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201799    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-run-systemd\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-run-ovn\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5f6f\" (UniqueName: \"kubernetes.io/projected/2e309c4f-2923-4a09-9b32-94a389560a90-kube-api-access-l5f6f\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-multus-conf-dir\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08c98ae9-ea2f-4de5-9b8b-06cc63a945d7-hosts-file\") pod \"node-resolver-vqw9q\" (UID: \"08c98ae9-ea2f-4de5-9b8b-06cc63a945d7\") " pod="openshift-dns/node-resolver-vqw9q"
Jul 16 20:00:24.201955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97cc28a3-e813-4407-bad7-8f53dec993f4-serviceca\") pod \"node-ca-5h765\" (UID: \"97cc28a3-e813-4407-bad7-8f53dec993f4\") " pod="openshift-image-registry/node-ca-5h765"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.201976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8v4\" (UniqueName: \"kubernetes.io/projected/0238483d-389c-44cf-92b1-7f8cf040159e-kube-api-access-hs8v4\") pod \"network-metrics-daemon-6ffx6\" (UID: \"0238483d-389c-44cf-92b1-7f8cf040159e\") " pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c-agent-certs\") pod \"konnectivity-agent-m74x5\" (UID: \"e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c\") " pod="kube-system/konnectivity-agent-m74x5"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202037    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-cni-netd\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-multus-cni-dir\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-os-release\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-var-lib-cni-bin\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45n6q\" (UniqueName: \"kubernetes.io/projected/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-kube-api-access-45n6q\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/29d77657-97fd-4e5a-97d9-63dea83fadf9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202231    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcv7f\" (UniqueName: \"kubernetes.io/projected/d9657416-219b-4d50-a3ff-7db1065072aa-kube-api-access-vcv7f\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2e309c4f-2923-4a09-9b32-94a389560a90-ovnkube-config\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-etc-selinux\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202432    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-etc-selinux\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202512    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-run-ovn\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202764    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e309c4f-2923-4a09-9b32-94a389560a90-ovn-node-metrics-cert\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.202936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.202903    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-cni-netd\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.203954 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.203001    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2e309c4f-2923-4a09-9b32-94a389560a90-ovnkube-script-lib\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.203954 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.203114    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-run-systemd\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.203954 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.203250    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-host-cni-bin\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.203954 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.203325    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-device-dir\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.203954 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.203346    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.203954 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.203373    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e309c4f-2923-4a09-9b32-94a389560a90-etc-openvswitch\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.203954 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.203495    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2e309c4f-2923-4a09-9b32-94a389560a90-ovnkube-config\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.205703 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.205677    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c-agent-certs\") pod \"konnectivity-agent-m74x5\" (UID: \"e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c\") " pod="kube-system/konnectivity-agent-m74x5"
Jul 16 20:00:24.209010 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.208987    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwg7d\" (UniqueName: \"kubernetes.io/projected/13867e43-5cc6-4cc3-8def-bb0ed6115cbc-kube-api-access-hwg7d\") pod \"aws-ebs-csi-driver-node-jdrrx\" (UID: \"13867e43-5cc6-4cc3-8def-bb0ed6115cbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.211323 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.211305    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5f6f\" (UniqueName: \"kubernetes.io/projected/2e309c4f-2923-4a09-9b32-94a389560a90-kube-api-access-l5f6f\") pod \"ovnkube-node-srq56\" (UID: \"2e309c4f-2923-4a09-9b32-94a389560a90\") " pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.244926 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.244878    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-109.ec2.internal" event={"ID":"93b88c9554c3d1c5c57b49c2ee4f8f5e","Type":"ContainerStarted","Data":"f2b0c6c542b725450b5abc57be8ecc67f11c67eddf190b4563b11ec6a514e16d"}
Jul 16 20:00:24.246079 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.246045    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal" event={"ID":"6979e07120a9f54f8d29e294507849f7","Type":"ContainerStarted","Data":"da2b3609ff1854ffb76ed3261d7c1276a66eea75a88b11112b62602e0c545bac"}
Jul 16 20:00:24.302780 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.302692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-var-lib-kubelet\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.302780 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.302752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-etc-kubernetes\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.302780 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.302776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-system-cni-dir\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.303048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.302839    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-var-lib-kubelet\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.303048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.302841    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-system-cni-dir\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.303048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.302867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs\") pod \"network-metrics-daemon-6ffx6\" (UID: \"0238483d-389c-44cf-92b1-7f8cf040159e\") " pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:24.303048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.302907    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-cni-binary-copy\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.303048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.302931    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-multus-daemon-config\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.303048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.302839    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-etc-kubernetes\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.303048 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.302952    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:24.303048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.302999    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9711524-15cd-4165-854a-1453f37e4abc-dbus\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:24.303048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-run\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.303058    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs podName:0238483d-389c-44cf-92b1-7f8cf040159e nodeName:}" failed. No retries permitted until 2026-07-16 20:00:24.803035602 +0000 UTC m=+3.053656362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs") pod "network-metrics-daemon-6ffx6" (UID: "0238483d-389c-44cf-92b1-7f8cf040159e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303077    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-run\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-multus-socket-dir-parent\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303111    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-run-netns\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1df43134-62af-464b-9f40-304412852ec9-iptables-alerter-script\") pod \"iptables-alerter-gwntx\" (UID: \"1df43134-62af-464b-9f40-304412852ec9\") " pod="openshift-network-operator/iptables-alerter-gwntx"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29d77657-97fd-4e5a-97d9-63dea83fadf9-cni-binary-copy\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303189    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-multus-socket-dir-parent\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303192    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9711524-15cd-4165-854a-1453f37e4abc-kubelet-config\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9711524-15cd-4165-854a-1453f37e4abc-dbus\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303238    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9711524-15cd-4165-854a-1453f37e4abc-kubelet-config\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303205    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-run-netns\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv789\" (UniqueName: \"kubernetes.io/projected/1df43134-62af-464b-9f40-304412852ec9-kube-api-access-tv789\") pod \"iptables-alerter-gwntx\" (UID: \"1df43134-62af-464b-9f40-304412852ec9\") " pod="openshift-network-operator/iptables-alerter-gwntx"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c98ae9-ea2f-4de5-9b8b-06cc63a945d7-tmp-dir\") pod \"node-resolver-vqw9q\" (UID: \"08c98ae9-ea2f-4de5-9b8b-06cc63a945d7\") " pod="openshift-dns/node-resolver-vqw9q"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303339    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vclk\" (UniqueName: \"kubernetes.io/projected/08c98ae9-ea2f-4de5-9b8b-06cc63a945d7-kube-api-access-6vclk\") pod \"node-resolver-vqw9q\" (UID: \"08c98ae9-ea2f-4de5-9b8b-06cc63a945d7\") " pod="openshift-dns/node-resolver-vqw9q"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303374    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-sys\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97cc28a3-e813-4407-bad7-8f53dec993f4-host\") pod \"node-ca-5h765\" (UID: \"97cc28a3-e813-4407-bad7-8f53dec993f4\") " pod="openshift-image-registry/node-ca-5h765"
Jul 16 20:00:24.303505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkf46\" (UniqueName: \"kubernetes.io/projected/97cc28a3-e813-4407-bad7-8f53dec993f4-kube-api-access-mkf46\") pod \"node-ca-5h765\" (UID: \"97cc28a3-e813-4407-bad7-8f53dec993f4\") " pod="openshift-image-registry/node-ca-5h765"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-multus-conf-dir\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08c98ae9-ea2f-4de5-9b8b-06cc63a945d7-hosts-file\") pod \"node-resolver-vqw9q\" (UID: \"08c98ae9-ea2f-4de5-9b8b-06cc63a945d7\") " pod="openshift-dns/node-resolver-vqw9q"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303501    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97cc28a3-e813-4407-bad7-8f53dec993f4-serviceca\") pod \"node-ca-5h765\" (UID: \"97cc28a3-e813-4407-bad7-8f53dec993f4\") " pod="openshift-image-registry/node-ca-5h765"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8v4\" (UniqueName: \"kubernetes.io/projected/0238483d-389c-44cf-92b1-7f8cf040159e-kube-api-access-hs8v4\") pod \"network-metrics-daemon-6ffx6\" (UID: \"0238483d-389c-44cf-92b1-7f8cf040159e\") " pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-multus-cni-dir\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303566    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-cni-binary-copy\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-os-release\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-var-lib-cni-bin\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303609    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c98ae9-ea2f-4de5-9b8b-06cc63a945d7-tmp-dir\") pod \"node-resolver-vqw9q\" (UID: \"08c98ae9-ea2f-4de5-9b8b-06cc63a945d7\") " pod="openshift-dns/node-resolver-vqw9q"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45n6q\" (UniqueName: \"kubernetes.io/projected/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-kube-api-access-45n6q\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303631    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-sys\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303665    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-multus-daemon-config\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/29d77657-97fd-4e5a-97d9-63dea83fadf9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303694    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-multus-conf-dir\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcv7f\" (UniqueName: \"kubernetes.io/projected/d9657416-219b-4d50-a3ff-7db1065072aa-kube-api-access-vcv7f\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303761    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97cc28a3-e813-4407-bad7-8f53dec993f4-host\") pod \"node-ca-5h765\" (UID: \"97cc28a3-e813-4407-bad7-8f53dec993f4\") " pod="openshift-image-registry/node-ca-5h765"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303772    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08c98ae9-ea2f-4de5-9b8b-06cc63a945d7-hosts-file\") pod \"node-resolver-vqw9q\" (UID: \"08c98ae9-ea2f-4de5-9b8b-06cc63a945d7\") " pod="openshift-dns/node-resolver-vqw9q"
Jul 16 20:00:24.304473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-var-lib-kubelet\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303807    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1df43134-62af-464b-9f40-304412852ec9-host-slash\") pod \"iptables-alerter-gwntx\" (UID: \"1df43134-62af-464b-9f40-304412852ec9\") " pod="openshift-network-operator/iptables-alerter-gwntx"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303810    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1df43134-62af-464b-9f40-304412852ec9-iptables-alerter-script\") pod \"iptables-alerter-gwntx\" (UID: \"1df43134-62af-464b-9f40-304412852ec9\") " pod="openshift-network-operator/iptables-alerter-gwntx"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29d77657-97fd-4e5a-97d9-63dea83fadf9-cnibin\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303836    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-multus-cni-dir\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-os-release\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303882    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-var-lib-cni-bin\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303917    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1df43134-62af-464b-9f40-304412852ec9-host-slash\") pod \"iptables-alerter-gwntx\" (UID: \"1df43134-62af-464b-9f40-304412852ec9\") " pod="openshift-network-operator/iptables-alerter-gwntx"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303936    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-kubernetes\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303951    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29d77657-97fd-4e5a-97d9-63dea83fadf9-cni-binary-copy\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303958    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29d77657-97fd-4e5a-97d9-63dea83fadf9-cnibin\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303980    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-var-lib-kubelet\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.303884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-kubernetes\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.304026    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-sysctl-d\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-modprobe-d\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.305302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304092    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9657416-219b-4d50-a3ff-7db1065072aa-tmp\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.304107    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret podName:f9711524-15cd-4165-854a-1453f37e4abc nodeName:}" failed. No retries permitted until 2026-07-16 20:00:24.804089993 +0000 UTC m=+3.054710747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret") pod "global-pull-secret-syncer-szlh8" (UID: "f9711524-15cd-4165-854a-1453f37e4abc") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304216    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-sysctl-d\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304218    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-modprobe-d\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304239    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-hostroot\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304307    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d77657-97fd-4e5a-97d9-63dea83fadf9-system-cni-dir\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304317    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97cc28a3-e813-4407-bad7-8f53dec993f4-serviceca\") pod \"node-ca-5h765\" (UID: \"97cc28a3-e813-4407-bad7-8f53dec993f4\") " pod="openshift-image-registry/node-ca-5h765"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29d77657-97fd-4e5a-97d9-63dea83fadf9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-sysctl-conf\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304364    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-hostroot\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304367    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d77657-97fd-4e5a-97d9-63dea83fadf9-system-cni-dir\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304384    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-host\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304389    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/29d77657-97fd-4e5a-97d9-63dea83fadf9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9657416-219b-4d50-a3ff-7db1065072aa-etc-tuned\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/29d77657-97fd-4e5a-97d9-63dea83fadf9-os-release\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304455    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-host\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.305883 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpzjn\" (UniqueName: \"kubernetes.io/projected/29d77657-97fd-4e5a-97d9-63dea83fadf9-kube-api-access-cpzjn\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304501    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/29d77657-97fd-4e5a-97d9-63dea83fadf9-os-release\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-run-k8s-cni-cncf-io\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304519    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29d77657-97fd-4e5a-97d9-63dea83fadf9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304525    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-sysctl-conf\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304547    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-run-k8s-cni-cncf-io\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-var-lib-cni-multus\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-sysconfig\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-run-multus-certs\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-systemd\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304704    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-var-lib-cni-multus\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-lib-modules\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-cnibin\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304761    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-systemd\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304761    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-host-run-multus-certs\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304753    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-etc-sysconfig\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/29d77657-97fd-4e5a-97d9-63dea83fadf9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.306592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpskv\" (UniqueName: \"kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv\") pod \"network-check-target-7jgvl\" (UID: \"ac9ea87f-36ee-4233-8037-2e3802ae7ae6\") " pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:24.307360 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304849    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9657416-219b-4d50-a3ff-7db1065072aa-lib-modules\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.307360 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.304976    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-cnibin\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.307360 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.305174    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/29d77657-97fd-4e5a-97d9-63dea83fadf9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.307360 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.306759    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9657416-219b-4d50-a3ff-7db1065072aa-tmp\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.307360 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.307064    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9657416-219b-4d50-a3ff-7db1065072aa-etc-tuned\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.315403 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.315375    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv789\" (UniqueName: \"kubernetes.io/projected/1df43134-62af-464b-9f40-304412852ec9-kube-api-access-tv789\") pod \"iptables-alerter-gwntx\" (UID: \"1df43134-62af-464b-9f40-304412852ec9\") " pod="openshift-network-operator/iptables-alerter-gwntx"
Jul 16 20:00:24.315564 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.315548    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkf46\" (UniqueName: \"kubernetes.io/projected/97cc28a3-e813-4407-bad7-8f53dec993f4-kube-api-access-mkf46\") pod \"node-ca-5h765\" (UID: \"97cc28a3-e813-4407-bad7-8f53dec993f4\") " pod="openshift-image-registry/node-ca-5h765"
Jul 16 20:00:24.315620 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.315565    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vclk\" (UniqueName: \"kubernetes.io/projected/08c98ae9-ea2f-4de5-9b8b-06cc63a945d7-kube-api-access-6vclk\") pod \"node-resolver-vqw9q\" (UID: \"08c98ae9-ea2f-4de5-9b8b-06cc63a945d7\") " pod="openshift-dns/node-resolver-vqw9q"
Jul 16 20:00:24.320193 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.320175    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:00:24.320193 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.320195    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:00:24.320330 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.320206    2576 projected.go:196] Error preparing data for projected volume kube-api-access-rpskv for pod openshift-network-diagnostics/network-check-target-7jgvl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:24.320330 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.320252    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv podName:ac9ea87f-36ee-4233-8037-2e3802ae7ae6 nodeName:}" failed. No retries permitted until 2026-07-16 20:00:24.820239718 +0000 UTC m=+3.070860457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rpskv" (UniqueName: "kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv") pod "network-check-target-7jgvl" (UID: "ac9ea87f-36ee-4233-8037-2e3802ae7ae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:24.323018 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.322991    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45n6q\" (UniqueName: \"kubernetes.io/projected/d85ace9e-44b7-44f8-8ce3-f31ac3e41644-kube-api-access-45n6q\") pod \"multus-c5bsz\" (UID: \"d85ace9e-44b7-44f8-8ce3-f31ac3e41644\") " pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.323247 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.323229    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcv7f\" (UniqueName: \"kubernetes.io/projected/d9657416-219b-4d50-a3ff-7db1065072aa-kube-api-access-vcv7f\") pod \"tuned-rjqp9\" (UID: \"d9657416-219b-4d50-a3ff-7db1065072aa\") " pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.324493 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.324472    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8v4\" (UniqueName: \"kubernetes.io/projected/0238483d-389c-44cf-92b1-7f8cf040159e-kube-api-access-hs8v4\") pod \"network-metrics-daemon-6ffx6\" (UID: \"0238483d-389c-44cf-92b1-7f8cf040159e\") " pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:24.325271 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.325249    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpzjn\" (UniqueName: \"kubernetes.io/projected/29d77657-97fd-4e5a-97d9-63dea83fadf9-kube-api-access-cpzjn\") pod \"multus-additional-cni-plugins-4rwcb\" (UID: \"29d77657-97fd-4e5a-97d9-63dea83fadf9\") " pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.395054 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.395021    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m74x5"
Jul 16 20:00:24.409465 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.409442    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:24.418621 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.418593    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx"
Jul 16 20:00:24.423698 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.423673    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5h765"
Jul 16 20:00:24.430703 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.430675    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4rwcb"
Jul 16 20:00:24.437935 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.437915    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vqw9q"
Jul 16 20:00:24.443950 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.443912    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c5bsz"
Jul 16 20:00:24.449752 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:24.449694    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a3c35d_bfb2_4158_a8db_8b1d7f60c81c.slice/crio-5826ff459f3fc31e337061912fff68cc1dc20bacb622646167c0f57c181ceb16 WatchSource:0}: Error finding container 5826ff459f3fc31e337061912fff68cc1dc20bacb622646167c0f57c181ceb16: Status 404 returned error can't find the container with id 5826ff459f3fc31e337061912fff68cc1dc20bacb622646167c0f57c181ceb16
Jul 16 20:00:24.450075 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:24.450052    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cc28a3_e813_4407_bad7_8f53dec993f4.slice/crio-25cbc27328782d78e308a98a7f00d0c39a1a90de35f3ff7ebc736dbe7f94df96 WatchSource:0}: Error finding container 25cbc27328782d78e308a98a7f00d0c39a1a90de35f3ff7ebc736dbe7f94df96: Status 404 returned error can't find the container with id 25cbc27328782d78e308a98a7f00d0c39a1a90de35f3ff7ebc736dbe7f94df96
Jul 16 20:00:24.450813 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.450791    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gwntx"
Jul 16 20:00:24.451137 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:24.451024    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13867e43_5cc6_4cc3_8def_bb0ed6115cbc.slice/crio-e8d04332b1f4f5ba48a8e53e670d452958b9b36c9e8d0c28999f8700c1317d68 WatchSource:0}: Error finding container e8d04332b1f4f5ba48a8e53e670d452958b9b36c9e8d0c28999f8700c1317d68: Status 404 returned error can't find the container with id e8d04332b1f4f5ba48a8e53e670d452958b9b36c9e8d0c28999f8700c1317d68
Jul 16 20:00:24.451835 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:24.451588    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d77657_97fd_4e5a_97d9_63dea83fadf9.slice/crio-aeadbb6ce953e92c6fbb75826ef7e14da4c3e32661c5dcb5549c29e5e7b5879d WatchSource:0}: Error finding container aeadbb6ce953e92c6fbb75826ef7e14da4c3e32661c5dcb5549c29e5e7b5879d: Status 404 returned error can't find the container with id aeadbb6ce953e92c6fbb75826ef7e14da4c3e32661c5dcb5549c29e5e7b5879d
Jul 16 20:00:24.456891 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.456869    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rjqp9"
Jul 16 20:00:24.465816 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:24.465736    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e309c4f_2923_4a09_9b32_94a389560a90.slice/crio-280d594933bd5182c49ab33ca07a1d85badf30989d748b7205641d8397a190f5 WatchSource:0}: Error finding container 280d594933bd5182c49ab33ca07a1d85badf30989d748b7205641d8397a190f5: Status 404 returned error can't find the container with id 280d594933bd5182c49ab33ca07a1d85badf30989d748b7205641d8397a190f5
Jul 16 20:00:24.471944 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:24.471919    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c98ae9_ea2f_4de5_9b8b_06cc63a945d7.slice/crio-29d10931de0ad41a8ed6e5ed3b47c59b62fa8111f61154d49577e3b1ca23d631 WatchSource:0}: Error finding container 29d10931de0ad41a8ed6e5ed3b47c59b62fa8111f61154d49577e3b1ca23d631: Status 404 returned error can't find the container with id 29d10931de0ad41a8ed6e5ed3b47c59b62fa8111f61154d49577e3b1ca23d631
Jul 16 20:00:24.499694 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:24.499666    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df43134_62af_464b_9f40_304412852ec9.slice/crio-a94f1b9673b8f23364c03b8a73aeceb64890e96f5990ea101ce65abd15ed9e6c WatchSource:0}: Error finding container a94f1b9673b8f23364c03b8a73aeceb64890e96f5990ea101ce65abd15ed9e6c: Status 404 returned error can't find the container with id a94f1b9673b8f23364c03b8a73aeceb64890e96f5990ea101ce65abd15ed9e6c
Jul 16 20:00:24.503986 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:24.503250    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85ace9e_44b7_44f8_8ce3_f31ac3e41644.slice/crio-a0830e72a75717c480b66e4d0ef3c2acee49bde11b66d92169b72c3a8f297d8d WatchSource:0}: Error finding container a0830e72a75717c480b66e4d0ef3c2acee49bde11b66d92169b72c3a8f297d8d: Status 404 returned error can't find the container with id a0830e72a75717c480b66e4d0ef3c2acee49bde11b66d92169b72c3a8f297d8d
Jul 16 20:00:24.506274 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:24.506252    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9657416_219b_4d50_a3ff_7db1065072aa.slice/crio-a90b70b89deadc0572d5ff27842857a746282b11307a4dce3aa385f74b2879aa WatchSource:0}: Error finding container a90b70b89deadc0572d5ff27842857a746282b11307a4dce3aa385f74b2879aa: Status 404 returned error can't find the container with id a90b70b89deadc0572d5ff27842857a746282b11307a4dce3aa385f74b2879aa
Jul 16 20:00:24.808753 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.808701    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs\") pod \"network-metrics-daemon-6ffx6\" (UID: \"0238483d-389c-44cf-92b1-7f8cf040159e\") " pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:24.808937 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.808788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:24.808937 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.808904    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:24.808937 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.808913    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:24.809132 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.808980    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret podName:f9711524-15cd-4165-854a-1453f37e4abc nodeName:}" failed. No retries permitted until 2026-07-16 20:00:25.808960531 +0000 UTC m=+4.059581271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret") pod "global-pull-secret-syncer-szlh8" (UID: "f9711524-15cd-4165-854a-1453f37e4abc") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:24.809132 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.809003    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs podName:0238483d-389c-44cf-92b1-7f8cf040159e nodeName:}" failed. No retries permitted until 2026-07-16 20:00:25.808991339 +0000 UTC m=+4.059612083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs") pod "network-metrics-daemon-6ffx6" (UID: "0238483d-389c-44cf-92b1-7f8cf040159e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:24.909449 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:24.909405    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpskv\" (UniqueName: \"kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv\") pod \"network-check-target-7jgvl\" (UID: \"ac9ea87f-36ee-4233-8037-2e3802ae7ae6\") " pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:24.909678 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.909614    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:00:24.909678 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.909635    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:00:24.909678 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.909648    2576 projected.go:196] Error preparing data for projected volume kube-api-access-rpskv for pod openshift-network-diagnostics/network-check-target-7jgvl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:24.909883 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:24.909725    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv podName:ac9ea87f-36ee-4233-8037-2e3802ae7ae6 nodeName:}" failed. No retries permitted until 2026-07-16 20:00:25.909703751 +0000 UTC m=+4.160324490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rpskv" (UniqueName: "kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv") pod "network-check-target-7jgvl" (UID: "ac9ea87f-36ee-4233-8037-2e3802ae7ae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:25.137982 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.137854    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 19:55:24 +0000 UTC" deadline="2027-12-24 02:31:02.992165478 +0000 UTC"
Jul 16 20:00:25.137982 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.137914    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12606h30m37.854255581s"
Jul 16 20:00:25.239502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.238994    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:25.239502 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:25.239126    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:25.259937 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.259858    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5h765" event={"ID":"97cc28a3-e813-4407-bad7-8f53dec993f4","Type":"ContainerStarted","Data":"25cbc27328782d78e308a98a7f00d0c39a1a90de35f3ff7ebc736dbe7f94df96"}
Jul 16 20:00:25.263937 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.263791    2576 generic.go:356] "Generic (PLEG): container finished" podID="6979e07120a9f54f8d29e294507849f7" containerID="fa4c87de13860eccb24a85b6ed2c6ece28e555973a0fa8015d7a9daf835633e9" exitCode=0
Jul 16 20:00:25.263937 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.263889    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal" event={"ID":"6979e07120a9f54f8d29e294507849f7","Type":"ContainerDied","Data":"fa4c87de13860eccb24a85b6ed2c6ece28e555973a0fa8015d7a9daf835633e9"}
Jul 16 20:00:25.281510 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.279628    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gwntx" event={"ID":"1df43134-62af-464b-9f40-304412852ec9","Type":"ContainerStarted","Data":"a94f1b9673b8f23364c03b8a73aeceb64890e96f5990ea101ce65abd15ed9e6c"}
Jul 16 20:00:25.292488 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.292402    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vqw9q" event={"ID":"08c98ae9-ea2f-4de5-9b8b-06cc63a945d7","Type":"ContainerStarted","Data":"29d10931de0ad41a8ed6e5ed3b47c59b62fa8111f61154d49577e3b1ca23d631"}
Jul 16 20:00:25.305105 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.305016    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m74x5" event={"ID":"e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c","Type":"ContainerStarted","Data":"5826ff459f3fc31e337061912fff68cc1dc20bacb622646167c0f57c181ceb16"}
Jul 16 20:00:25.318545 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.318284    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rjqp9" event={"ID":"d9657416-219b-4d50-a3ff-7db1065072aa","Type":"ContainerStarted","Data":"a90b70b89deadc0572d5ff27842857a746282b11307a4dce3aa385f74b2879aa"}
Jul 16 20:00:25.340773 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.340669    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c5bsz" event={"ID":"d85ace9e-44b7-44f8-8ce3-f31ac3e41644","Type":"ContainerStarted","Data":"a0830e72a75717c480b66e4d0ef3c2acee49bde11b66d92169b72c3a8f297d8d"}
Jul 16 20:00:25.362210 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.362084    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srq56" event={"ID":"2e309c4f-2923-4a09-9b32-94a389560a90","Type":"ContainerStarted","Data":"280d594933bd5182c49ab33ca07a1d85badf30989d748b7205641d8397a190f5"}
Jul 16 20:00:25.376491 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.376404    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rwcb" event={"ID":"29d77657-97fd-4e5a-97d9-63dea83fadf9","Type":"ContainerStarted","Data":"aeadbb6ce953e92c6fbb75826ef7e14da4c3e32661c5dcb5549c29e5e7b5879d"}
Jul 16 20:00:25.383183 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.383116    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx" event={"ID":"13867e43-5cc6-4cc3-8def-bb0ed6115cbc","Type":"ContainerStarted","Data":"e8d04332b1f4f5ba48a8e53e670d452958b9b36c9e8d0c28999f8700c1317d68"}
Jul 16 20:00:25.820417 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.819589    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs\") pod \"network-metrics-daemon-6ffx6\" (UID: \"0238483d-389c-44cf-92b1-7f8cf040159e\") " pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:25.820417 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.819703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:25.820417 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:25.819856    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:25.820417 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:25.819922    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret podName:f9711524-15cd-4165-854a-1453f37e4abc nodeName:}" failed. No retries permitted until 2026-07-16 20:00:27.819902924 +0000 UTC m=+6.070523667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret") pod "global-pull-secret-syncer-szlh8" (UID: "f9711524-15cd-4165-854a-1453f37e4abc") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:25.820417 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:25.820336    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:25.820417 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:25.820383    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs podName:0238483d-389c-44cf-92b1-7f8cf040159e nodeName:}" failed. No retries permitted until 2026-07-16 20:00:27.820367498 +0000 UTC m=+6.070988241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs") pod "network-metrics-daemon-6ffx6" (UID: "0238483d-389c-44cf-92b1-7f8cf040159e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:25.921166 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:25.921108    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpskv\" (UniqueName: \"kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv\") pod \"network-check-target-7jgvl\" (UID: \"ac9ea87f-36ee-4233-8037-2e3802ae7ae6\") " pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:25.921356 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:25.921309    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:00:25.921356 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:25.921333    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:00:25.921356 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:25.921349    2576 projected.go:196] Error preparing data for projected volume kube-api-access-rpskv for pod openshift-network-diagnostics/network-check-target-7jgvl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:25.921542 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:25.921418    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv podName:ac9ea87f-36ee-4233-8037-2e3802ae7ae6 nodeName:}" failed. No retries permitted until 2026-07-16 20:00:27.921393777 +0000 UTC m=+6.172014525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rpskv" (UniqueName: "kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv") pod "network-check-target-7jgvl" (UID: "ac9ea87f-36ee-4233-8037-2e3802ae7ae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:26.117224 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:26.117139    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:00:26.138279 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:26.138207    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 19:55:24 +0000 UTC" deadline="2028-01-22 10:13:06.479293092 +0000 UTC"
Jul 16 20:00:26.138279 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:26.138246    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13310h12m40.341052501s"
Jul 16 20:00:26.240425 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:26.239904    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:26.240425 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:26.240050    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:26.240982 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:26.240803    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:26.240982 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:26.240915    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:26.416770 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:26.416672    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal" event={"ID":"6979e07120a9f54f8d29e294507849f7","Type":"ContainerStarted","Data":"8fe4a0ff458a613c4bf4fa803648b86e74735c819f7f9fcf4b156fc08e13050f"}
Jul 16 20:00:27.239990 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:27.239401    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:27.239990 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:27.239557    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:27.841179 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:27.841115    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs\") pod \"network-metrics-daemon-6ffx6\" (UID: \"0238483d-389c-44cf-92b1-7f8cf040159e\") " pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:27.841360 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:27.841188    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:27.841360 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:27.841318    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:27.841461 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:27.841387    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret podName:f9711524-15cd-4165-854a-1453f37e4abc nodeName:}" failed. No retries permitted until 2026-07-16 20:00:31.841366657 +0000 UTC m=+10.091987414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret") pod "global-pull-secret-syncer-szlh8" (UID: "f9711524-15cd-4165-854a-1453f37e4abc") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:27.841852 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:27.841828    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:27.841975 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:27.841887    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs podName:0238483d-389c-44cf-92b1-7f8cf040159e nodeName:}" failed. No retries permitted until 2026-07-16 20:00:31.841870966 +0000 UTC m=+10.092491707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs") pod "network-metrics-daemon-6ffx6" (UID: "0238483d-389c-44cf-92b1-7f8cf040159e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:27.942517 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:27.942358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpskv\" (UniqueName: \"kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv\") pod \"network-check-target-7jgvl\" (UID: \"ac9ea87f-36ee-4233-8037-2e3802ae7ae6\") " pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:27.942736 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:27.942539    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:00:27.942736 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:27.942562    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:00:27.942736 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:27.942576    2576 projected.go:196] Error preparing data for projected volume kube-api-access-rpskv for pod openshift-network-diagnostics/network-check-target-7jgvl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:27.942736 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:27.942637    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv podName:ac9ea87f-36ee-4233-8037-2e3802ae7ae6 nodeName:}" failed. No retries permitted until 2026-07-16 20:00:31.942618537 +0000 UTC m=+10.193239294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rpskv" (UniqueName: "kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv") pod "network-check-target-7jgvl" (UID: "ac9ea87f-36ee-4233-8037-2e3802ae7ae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:28.242041 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:28.241950    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:28.242479 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:28.242116    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:28.242479 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:28.242203    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:28.242479 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:28.242284    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:29.239465 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:29.239428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:29.239667 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:29.239578    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:30.241261 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:30.240620    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:30.241261 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:30.240753    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:30.241261 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:30.241122    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:30.241261 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:30.241205    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:31.238583 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:31.238508    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:31.238800 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:31.238727    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:31.876351 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:31.876303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs\") pod \"network-metrics-daemon-6ffx6\" (UID: \"0238483d-389c-44cf-92b1-7f8cf040159e\") " pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:31.876869 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:31.876402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:31.876869 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:31.876561    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:31.876869 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:31.876638    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret podName:f9711524-15cd-4165-854a-1453f37e4abc nodeName:}" failed. No retries permitted until 2026-07-16 20:00:39.87661362 +0000 UTC m=+18.127234361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret") pod "global-pull-secret-syncer-szlh8" (UID: "f9711524-15cd-4165-854a-1453f37e4abc") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:31.877164 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:31.877139    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:31.877234 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:31.877212    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs podName:0238483d-389c-44cf-92b1-7f8cf040159e nodeName:}" failed. No retries permitted until 2026-07-16 20:00:39.877193442 +0000 UTC m=+18.127814183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs") pod "network-metrics-daemon-6ffx6" (UID: "0238483d-389c-44cf-92b1-7f8cf040159e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:31.977121 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:31.977084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpskv\" (UniqueName: \"kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv\") pod \"network-check-target-7jgvl\" (UID: \"ac9ea87f-36ee-4233-8037-2e3802ae7ae6\") " pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:31.977332 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:31.977303    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:00:31.977408 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:31.977344    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:00:31.977408 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:31.977359    2576 projected.go:196] Error preparing data for projected volume kube-api-access-rpskv for pod openshift-network-diagnostics/network-check-target-7jgvl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:31.977517 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:31.977432    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv podName:ac9ea87f-36ee-4233-8037-2e3802ae7ae6 nodeName:}" failed. No retries permitted until 2026-07-16 20:00:39.977409653 +0000 UTC m=+18.228030407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rpskv" (UniqueName: "kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv") pod "network-check-target-7jgvl" (UID: "ac9ea87f-36ee-4233-8037-2e3802ae7ae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:32.240665 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:32.240014    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:32.240665 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:32.240147    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:32.240665 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:32.240487    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:32.240665 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:32.240572    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:33.239027 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:33.238487    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:33.239027 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:33.238648    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:34.239552 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:34.239514    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:34.240071 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:34.239650    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:34.240149 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:34.240074    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:34.240207 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:34.240190    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:35.238971 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:35.238931    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:35.239146 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:35.239051    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:36.239466 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:36.239432    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:36.239926 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:36.239583    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:36.239926 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:36.239620    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:36.239926 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:36.239750    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:37.239574 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:37.239492    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:37.239951 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:37.239620    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:38.239427 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:38.239388    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:38.239599 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:38.239434    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:38.239599 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:38.239521    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:38.239599 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:38.239590    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:39.239199 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:39.239154    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:39.239495 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:39.239289    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:39.941582 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:39.941544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:39.942092 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:39.941638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs\") pod \"network-metrics-daemon-6ffx6\" (UID: \"0238483d-389c-44cf-92b1-7f8cf040159e\") " pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:39.942092 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:39.941668    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:39.942092 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:39.941753    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret podName:f9711524-15cd-4165-854a-1453f37e4abc nodeName:}" failed. No retries permitted until 2026-07-16 20:00:55.941738036 +0000 UTC m=+34.192358773 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret") pod "global-pull-secret-syncer-szlh8" (UID: "f9711524-15cd-4165-854a-1453f37e4abc") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:39.942092 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:39.941759    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:39.942092 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:39.941807    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs podName:0238483d-389c-44cf-92b1-7f8cf040159e nodeName:}" failed. No retries permitted until 2026-07-16 20:00:55.941789162 +0000 UTC m=+34.192409898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs") pod "network-metrics-daemon-6ffx6" (UID: "0238483d-389c-44cf-92b1-7f8cf040159e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:40.042638 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:40.042609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpskv\" (UniqueName: \"kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv\") pod \"network-check-target-7jgvl\" (UID: \"ac9ea87f-36ee-4233-8037-2e3802ae7ae6\") " pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:40.042827 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:40.042766    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:00:40.042827 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:40.042782    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:00:40.042827 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:40.042795    2576 projected.go:196] Error preparing data for projected volume kube-api-access-rpskv for pod openshift-network-diagnostics/network-check-target-7jgvl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:40.042962 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:40.042847    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv podName:ac9ea87f-36ee-4233-8037-2e3802ae7ae6 nodeName:}" failed. No retries permitted until 2026-07-16 20:00:56.042832309 +0000 UTC m=+34.293453048 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rpskv" (UniqueName: "kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv") pod "network-check-target-7jgvl" (UID: "ac9ea87f-36ee-4233-8037-2e3802ae7ae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:40.238809 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:40.238720    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:40.238809 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:40.238758    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:40.239034 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:40.238868    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:40.239088 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:40.239026    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:41.239350 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:41.239314    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:41.239689 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:41.239424    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:42.239496 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:42.239456    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:42.239496 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:42.239481    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:42.240020 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:42.239593    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:42.240020 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:42.239663    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:43.238560 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:43.238524    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:43.238786 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:43.238663    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:44.239509 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:44.239036    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:44.239509 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:44.239191    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:44.239509 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:44.239245    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:44.239509 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:44.239377    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:45.238890 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:45.238856    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:45.239062 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:45.238988    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:46.239328 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.238993    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:46.240194 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.238994    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:46.240194 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:46.239449    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:46.240194 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:46.239523    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:46.465098 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.465015    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-109.ec2.internal" event={"ID":"93b88c9554c3d1c5c57b49c2ee4f8f5e","Type":"ContainerStarted","Data":"bc5068544ce3bad3a116bee66fd29b68b135a3e9842c341c252ec110cd689691"}
Jul 16 20:00:46.466898 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.466867    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vqw9q" event={"ID":"08c98ae9-ea2f-4de5-9b8b-06cc63a945d7","Type":"ContainerStarted","Data":"f85027eb9a7c30630d4d8c3a79288e4d3577c20d2c0ee9d3f7afccf78f58813d"}
Jul 16 20:00:46.468422 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.468388    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m74x5" event={"ID":"e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c","Type":"ContainerStarted","Data":"f307a39abc4b48d5a48dcf5170c3316629c6430bf1669191b62aa8b146faa4e1"}
Jul 16 20:00:46.470272 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.470241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rjqp9" event={"ID":"d9657416-219b-4d50-a3ff-7db1065072aa","Type":"ContainerStarted","Data":"f8db3074b0026ab46936a25827446051a0bea0f75b3ea5cb735073e91f19af3e"}
Jul 16 20:00:46.471820 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.471796    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c5bsz" event={"ID":"d85ace9e-44b7-44f8-8ce3-f31ac3e41644","Type":"ContainerStarted","Data":"8430118ab3759cdfc7c19d9448e9363ed9e786189111dbb7db6f81ff2b41be99"}
Jul 16 20:00:46.475094 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.475061    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srq56" event={"ID":"2e309c4f-2923-4a09-9b32-94a389560a90","Type":"ContainerStarted","Data":"fe9f3be4fe5777e520c9075330bcc120dcb26d4d16828f2fb93fad94980f293d"}
Jul 16 20:00:46.475228 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.475140    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srq56" event={"ID":"2e309c4f-2923-4a09-9b32-94a389560a90","Type":"ContainerStarted","Data":"9f2ab60f6e139bc5588f64148773ed6414c268a96cfd336e2bdf13edd1d2db86"}
Jul 16 20:00:46.475228 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.475180    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srq56" event={"ID":"2e309c4f-2923-4a09-9b32-94a389560a90","Type":"ContainerStarted","Data":"185ae80a3de6d7ed4e1f0a6fe0c023db87449869e81f6450422b8b340c6630d9"}
Jul 16 20:00:46.475228 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.475227    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srq56" event={"ID":"2e309c4f-2923-4a09-9b32-94a389560a90","Type":"ContainerStarted","Data":"c0f1bc3a9a89a9c004edfe3d1bd06e8142061c9918795aec78583ac2afc2e5e1"}
Jul 16 20:00:46.475357 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.475269    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srq56" event={"ID":"2e309c4f-2923-4a09-9b32-94a389560a90","Type":"ContainerStarted","Data":"f43b4c490d9f49a7e3587a7ec439741044688a7e91bfe51c0dadc241d0640c74"}
Jul 16 20:00:46.475357 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.475318    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srq56" event={"ID":"2e309c4f-2923-4a09-9b32-94a389560a90","Type":"ContainerStarted","Data":"7db333c36cf2c984eee55e2d7a02d9ed4b914df8a45b9a8cbe01688fbae89599"}
Jul 16 20:00:46.476652 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.476625    2576 generic.go:356] "Generic (PLEG): container finished" podID="29d77657-97fd-4e5a-97d9-63dea83fadf9" containerID="51f00ba86b27ebb2f14b09dfaf37274911514af3a2ba8211ed42865861b13ed0" exitCode=0
Jul 16 20:00:46.476779 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.476693    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rwcb" event={"ID":"29d77657-97fd-4e5a-97d9-63dea83fadf9","Type":"ContainerDied","Data":"51f00ba86b27ebb2f14b09dfaf37274911514af3a2ba8211ed42865861b13ed0"}
Jul 16 20:00:46.478552 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.478530    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx" event={"ID":"13867e43-5cc6-4cc3-8def-bb0ed6115cbc","Type":"ContainerStarted","Data":"b8e1e552c74ecfe524e4503240b8ef0fe0fa71bbc2ab08c151a596ad8371874f"}
Jul 16 20:00:46.480900 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.480846    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-109.ec2.internal" podStartSLOduration=23.480829797 podStartE2EDuration="23.480829797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:00:26.43427438 +0000 UTC m=+4.684895143" watchObservedRunningTime="2026-07-16 20:00:46.480829797 +0000 UTC m=+24.731450559"
Jul 16 20:00:46.483471 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.483449    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5h765" event={"ID":"97cc28a3-e813-4407-bad7-8f53dec993f4","Type":"ContainerStarted","Data":"31819390a9a7ed50d639ccdbc15f53c06d7ba422c541d4314ebf75fddcb8ccfd"}
Jul 16 20:00:46.505958 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.505918    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-109.ec2.internal" podStartSLOduration=23.505902134 podStartE2EDuration="23.505902134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:00:46.480251369 +0000 UTC m=+24.730872128" watchObservedRunningTime="2026-07-16 20:00:46.505902134 +0000 UTC m=+24.756522892"
Jul 16 20:00:46.526501 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.526459    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c5bsz" podStartSLOduration=2.549835592 podStartE2EDuration="23.52644262s" totalImagesPullingTime="20.976607028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:00:46.525932369 +0000 UTC m=+24.776553129" watchObservedRunningTime="2026-07-16 20:00:46.52644262 +0000 UTC m=+24.777063379"
Jul 16 20:00:46.543931 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.543890    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rjqp9" podStartSLOduration=2.615615899 podStartE2EDuration="23.543875398s" totalImagesPullingTime="20.928259499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:00:46.543640802 +0000 UTC m=+24.794261564" watchObservedRunningTime="2026-07-16 20:00:46.543875398 +0000 UTC m=+24.794496157"
Jul 16 20:00:46.559103 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.559064    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m74x5" podStartSLOduration=11.680499287 podStartE2EDuration="24.559050622s" totalImagesPullingTime="12.878551335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:00:46.55884361 +0000 UTC m=+24.809464369" watchObservedRunningTime="2026-07-16 20:00:46.559050622 +0000 UTC m=+24.809671378"
Jul 16 20:00:46.572454 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.572416    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vqw9q" podStartSLOduration=2.653715081 podStartE2EDuration="23.572405041s" totalImagesPullingTime="20.91868996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:00:46.572142171 +0000 UTC m=+24.822762930" watchObservedRunningTime="2026-07-16 20:00:46.572405041 +0000 UTC m=+24.823025800"
Jul 16 20:00:46.587589 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:46.587550    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5h765" podStartSLOduration=2.6245897769999997 podStartE2EDuration="23.587541211s" totalImagesPullingTime="20.962951434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:00:46.58719239 +0000 UTC m=+24.837813150" watchObservedRunningTime="2026-07-16 20:00:46.587541211 +0000 UTC m=+24.838161970"
Jul 16 20:00:47.026615 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:47.026592    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 20:00:47.164481 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:47.164344    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T20:00:47.0266118Z","UUID":"a225aae4-c4bb-4fc4-b9a2-b5a7eec1ffb4","Handler":null,"Name":"","Endpoint":""}
Jul 16 20:00:47.165853 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:47.165834    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 20:00:47.165972 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:47.165870    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 20:00:47.239397 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:47.239365    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:47.239842 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:47.239486    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:47.487593 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:47.487493    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gwntx" event={"ID":"1df43134-62af-464b-9f40-304412852ec9","Type":"ContainerStarted","Data":"47204d9594516e95328423937e04fe455c51cfe33b90db51c9bd104c91303b90"}
Jul 16 20:00:47.490993 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:47.490967    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx" event={"ID":"13867e43-5cc6-4cc3-8def-bb0ed6115cbc","Type":"ContainerStarted","Data":"39b2a22f4095ec78afb902fe21350d8341f748438e65762f84d6cbf5191699f7"}
Jul 16 20:00:47.507917 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:47.507865    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gwntx" podStartSLOduration=3.590101051 podStartE2EDuration="24.507844726s" totalImagesPullingTime="20.917743675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:00:47.506849673 +0000 UTC m=+25.757470433" watchObservedRunningTime="2026-07-16 20:00:47.507844726 +0000 UTC m=+25.758465486"
Jul 16 20:00:48.239018 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:48.238951    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:48.239158 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:48.239102    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:48.239158 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:48.239139    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:48.239265 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:48.239247    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:48.496348 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:48.495985    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srq56" event={"ID":"2e309c4f-2923-4a09-9b32-94a389560a90","Type":"ContainerStarted","Data":"765507e77d3fdd2b4c9c17997d2fbbcb297ccdffca96da711279fa7519b7067d"}
Jul 16 20:00:48.498530 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:48.498492    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx" event={"ID":"13867e43-5cc6-4cc3-8def-bb0ed6115cbc","Type":"ContainerStarted","Data":"114e01c22e5cf62f15a9daf99e3980826d017a4f67d6957d5fb3e9084336636d"}
Jul 16 20:00:48.517766 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:48.517712    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jdrrx" podStartSLOduration=3.340018123 podStartE2EDuration="26.517677663s" totalImagesPullingTime="23.17765954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:00:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:00:48.51758774 +0000 UTC m=+26.768208498" watchObservedRunningTime="2026-07-16 20:00:48.517677663 +0000 UTC m=+26.768298423"
Jul 16 20:00:49.239302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:49.239269    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:49.239668 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:49.239394    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:50.239477 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:50.239395    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:50.240122 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:50.239534    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:50.240122 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:50.239585    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:50.240122 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:50.239727    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:51.239297 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:51.239123    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:51.239432 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:51.239381    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:51.464079 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:51.464005    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m74x5"
Jul 16 20:00:51.464805 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:51.464541    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m74x5"
Jul 16 20:00:51.509819 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:51.509789    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srq56" event={"ID":"2e309c4f-2923-4a09-9b32-94a389560a90","Type":"ContainerStarted","Data":"2dca07ee01954a52fc8cfd65512e92deee8ab35f22cf6828d2e5ff91d4dd2d95"}
Jul 16 20:00:51.510037 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:51.510019    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:51.515888 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:51.515856    2576 generic.go:356] "Generic (PLEG): container finished" podID="29d77657-97fd-4e5a-97d9-63dea83fadf9" containerID="79e3da678b0db96b5d6b5a5298cd71cda8f5c66bdb33c576314bcfb8e3abae48" exitCode=0
Jul 16 20:00:51.516031 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:51.515948    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rwcb" event={"ID":"29d77657-97fd-4e5a-97d9-63dea83fadf9","Type":"ContainerDied","Data":"79e3da678b0db96b5d6b5a5298cd71cda8f5c66bdb33c576314bcfb8e3abae48"}
Jul 16 20:00:51.516088 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:51.516038    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m74x5"
Jul 16 20:00:51.516516 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:51.516496    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m74x5"
Jul 16 20:00:51.526945 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:51.526924    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:51.543523 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:51.543484    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-srq56" podStartSLOduration=8.525940607 podStartE2EDuration="29.54347139s" totalImagesPullingTime="21.017530783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:00:51.543286918 +0000 UTC m=+29.793907676" watchObservedRunningTime="2026-07-16 20:00:51.54347139 +0000 UTC m=+29.794092149"
Jul 16 20:00:52.240170 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:52.240134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:52.240427 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:52.240259    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:52.240732 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:52.240710    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:52.240814 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:52.240800    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:52.518624 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:52.518554    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:52.519040 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:52.518631    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:52.534783 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:52.534755    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:00:52.925898 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:52.925860    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6ffx6"]
Jul 16 20:00:52.926097 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:52.926042    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:52.926229 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:52.926208    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:52.933064 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:52.933012    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7jgvl"]
Jul 16 20:00:52.933176 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:52.933138    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:52.933258 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:52.933240    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:52.933853 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:52.933831    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-szlh8"]
Jul 16 20:00:52.933959 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:52.933924    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:52.934025 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:52.934004    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:54.239065 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:54.239035    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:54.239458 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:54.239033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:54.239458 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:54.239149    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:54.239458 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:54.239033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:54.239458 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:54.239216    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:54.239458 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:54.239284    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:55.962659 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:55.962475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs\") pod \"network-metrics-daemon-6ffx6\" (UID: \"0238483d-389c-44cf-92b1-7f8cf040159e\") " pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:55.963229 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:55.962678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:55.963229 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:55.962615    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:55.963229 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:55.962786    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:55.963229 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:55.962793    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs podName:0238483d-389c-44cf-92b1-7f8cf040159e nodeName:}" failed. No retries permitted until 2026-07-16 20:01:27.962769504 +0000 UTC m=+66.213390256 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs") pod "network-metrics-daemon-6ffx6" (UID: "0238483d-389c-44cf-92b1-7f8cf040159e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:00:55.963229 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:55.962824    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret podName:f9711524-15cd-4165-854a-1453f37e4abc nodeName:}" failed. No retries permitted until 2026-07-16 20:01:27.962812903 +0000 UTC m=+66.213433640 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret") pod "global-pull-secret-syncer-szlh8" (UID: "f9711524-15cd-4165-854a-1453f37e4abc") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:00:56.063352 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:56.063317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpskv\" (UniqueName: \"kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv\") pod \"network-check-target-7jgvl\" (UID: \"ac9ea87f-36ee-4233-8037-2e3802ae7ae6\") " pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:56.063499 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:56.063479    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:00:56.063499 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:56.063496    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:00:56.063581 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:56.063505    2576 projected.go:196] Error preparing data for projected volume kube-api-access-rpskv for pod openshift-network-diagnostics/network-check-target-7jgvl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:56.063581 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:56.063551    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv podName:ac9ea87f-36ee-4233-8037-2e3802ae7ae6 nodeName:}" failed. No retries permitted until 2026-07-16 20:01:28.063537959 +0000 UTC m=+66.314158696 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rpskv" (UniqueName: "kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv") pod "network-check-target-7jgvl" (UID: "ac9ea87f-36ee-4233-8037-2e3802ae7ae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:00:56.239353 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:56.239275    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:56.239506 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:56.239278    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:56.239506 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:56.239401    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlh8" podUID="f9711524-15cd-4165-854a-1453f37e4abc"
Jul 16 20:00:56.239506 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:56.239450    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:56.239649 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:56.239532    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6ffx6" podUID="0238483d-389c-44cf-92b1-7f8cf040159e"
Jul 16 20:00:56.239649 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:56.239597    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jgvl" podUID="ac9ea87f-36ee-4233-8037-2e3802ae7ae6"
Jul 16 20:00:56.528968 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:56.528891    2576 generic.go:356] "Generic (PLEG): container finished" podID="29d77657-97fd-4e5a-97d9-63dea83fadf9" containerID="4c27c1efdd4688fca57c0e8519a74e5716d93541e8a5a174e872575b2b8ccd92" exitCode=0
Jul 16 20:00:56.528968 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:56.528927    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rwcb" event={"ID":"29d77657-97fd-4e5a-97d9-63dea83fadf9","Type":"ContainerDied","Data":"4c27c1efdd4688fca57c0e8519a74e5716d93541e8a5a174e872575b2b8ccd92"}
Jul 16 20:00:58.133128 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.133054    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vqw9q_08c98ae9-ea2f-4de5-9b8b-06cc63a945d7/dns-node-resolver/0.log"
Jul 16 20:00:58.133577 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.133269    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-109.ec2.internal" event="NodeReady"
Jul 16 20:00:58.133577 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.133398    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 20:00:58.169256 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.169225    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-ff7887449-bq987"]
Jul 16 20:00:58.171900 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.171883    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.174439 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.174419    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 20:00:58.174524 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.174425    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vjpvn\""
Jul 16 20:00:58.175046 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.175028    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 20:00:58.175119 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.175059    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 20:00:58.181113 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.181092    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 20:00:58.182819 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.182801    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l4947"]
Jul 16 20:00:58.185480 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.185464    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-gwdvz"]
Jul 16 20:00:58.185603 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.185587    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l4947"
Jul 16 20:00:58.188148 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.188132    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-gwdvz"
Jul 16 20:00:58.189610 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.189592    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mtlsx\""
Jul 16 20:00:58.189610 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.189603    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 20:00:58.189998 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.189972    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 20:00:58.190774 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.190753    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 16 20:00:58.191013 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.190980    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 16 20:00:58.191101 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.191047    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 16 20:00:58.191152 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.191133    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9lfk8\""
Jul 16 20:00:58.191238 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.191223    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 16 20:00:58.191528 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.191511    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 20:00:58.192810 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.192760    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ff7887449-bq987"]
Jul 16 20:00:58.193293 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.193277    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s4lrz"]
Jul 16 20:00:58.196286 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.196268    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.198711 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.198674    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 20:00:58.198804 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.198779    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 20:00:58.198865 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.198800    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 20:00:58.198865 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.198854    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-92cdv\""
Jul 16 20:00:58.198970 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.198909    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 20:00:58.201865 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.201846    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l4947"]
Jul 16 20:00:58.202502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.202487    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-gwdvz"]
Jul 16 20:00:58.206637 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.206618    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4lrz"]
Jul 16 20:00:58.207175 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.207161    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l4b9r"]
Jul 16 20:00:58.209945 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.209930    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:58.212111 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.212094    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 20:00:58.212198 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.212185    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 20:00:58.212586 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.212571    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fkb6w\""
Jul 16 20:00:58.220312 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.220295    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l4b9r"]
Jul 16 20:00:58.242112 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.242094    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:00:58.242209 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.242194    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:00:58.242388 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.242362    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:00:58.244447 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.244428    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 20:00:58.244537 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.244506    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfzlm\""
Jul 16 20:00:58.244674 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.244660    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 20:00:58.244959 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.244943    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 20:00:58.245060 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.245045    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 20:00:58.245103 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.245045    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nxdf6\""
Jul 16 20:00:58.279855 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.279832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-installation-pull-secrets\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.279969 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.279858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb00e945-1b1b-4387-8834-cd0365bdcd23-cert\") pod \"ingress-canary-l4947\" (UID: \"bb00e945-1b1b-4387-8834-cd0365bdcd23\") " pod="openshift-ingress-canary/ingress-canary-l4947"
Jul 16 20:00:58.279969 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.279876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nh5x\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-kube-api-access-7nh5x\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.279969 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.279904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6-signing-key\") pod \"service-ca-58598695f7-gwdvz\" (UID: \"3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6\") " pod="openshift-service-ca/service-ca-58598695f7-gwdvz"
Jul 16 20:00:58.279969 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.279952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-trusted-ca\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.280112 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.279999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.280112 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.280039    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.280112 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.280065    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5jh7\" (UniqueName: \"kubernetes.io/projected/bb00e945-1b1b-4387-8834-cd0365bdcd23-kube-api-access-k5jh7\") pod \"ingress-canary-l4947\" (UID: \"bb00e945-1b1b-4387-8834-cd0365bdcd23\") " pod="openshift-ingress-canary/ingress-canary-l4947"
Jul 16 20:00:58.280112 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.280085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-image-registry-private-configuration\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.280112 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.280101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-ca-trust-extracted\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.280276 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.280115    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbt4b\" (UniqueName: \"kubernetes.io/projected/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-api-access-lbt4b\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.280276 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.280164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6-signing-cabundle\") pod \"service-ca-58598695f7-gwdvz\" (UID: \"3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6\") " pod="openshift-service-ca/service-ca-58598695f7-gwdvz"
Jul 16 20:00:58.280276 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.280187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrrmf\" (UniqueName: \"kubernetes.io/projected/3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6-kube-api-access-rrrmf\") pod \"service-ca-58598695f7-gwdvz\" (UID: \"3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6\") " pod="openshift-service-ca/service-ca-58598695f7-gwdvz"
Jul 16 20:00:58.280276 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.280209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-crio-socket\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.280276 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.280231    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-data-volume\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.280276 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.280266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-bound-sa-token\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.280470 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.280286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.280470 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.280305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-certificates\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.380569 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.380537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-image-registry-private-configuration\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.380569 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.380574    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-ca-trust-extracted\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.380766 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.380590    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbt4b\" (UniqueName: \"kubernetes.io/projected/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-api-access-lbt4b\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.380766 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.380644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6-signing-cabundle\") pod \"service-ca-58598695f7-gwdvz\" (UID: \"3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6\") " pod="openshift-service-ca/service-ca-58598695f7-gwdvz"
Jul 16 20:00:58.380766 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.380672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrrmf\" (UniqueName: \"kubernetes.io/projected/3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6-kube-api-access-rrrmf\") pod \"service-ca-58598695f7-gwdvz\" (UID: \"3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6\") " pod="openshift-service-ca/service-ca-58598695f7-gwdvz"
Jul 16 20:00:58.380766 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.380688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-crio-socket\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.380766 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.380726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-data-volume\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.380766 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.380748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-bound-sa-token\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.381023 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.380915    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.381023 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.380948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59179dd-36c5-404f-837f-8e54b1c799ed-tmp-dir\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:58.381023 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.380982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-certificates\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.381023 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.380986    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:00:58.381023 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381014    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-crio-socket\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.381247 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-installation-pull-secrets\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.381247 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb00e945-1b1b-4387-8834-cd0365bdcd23-cert\") pod \"ingress-canary-l4947\" (UID: \"bb00e945-1b1b-4387-8834-cd0365bdcd23\") " pod="openshift-ingress-canary/ingress-canary-l4947"
Jul 16 20:00:58.381247 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.381134    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm podName:13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1 nodeName:}" failed. No retries permitted until 2026-07-16 20:00:58.881109657 +0000 UTC m=+37.131730415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4lrz" (UID: "13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1") : configmap "kube-rbac-proxy" not found
Jul 16 20:00:58.381247 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381170    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-data-volume\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.381247 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:58.381482 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nh5x\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-kube-api-access-7nh5x\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.381482 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xccl5\" (UniqueName: \"kubernetes.io/projected/f59179dd-36c5-404f-837f-8e54b1c799ed-kube-api-access-xccl5\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:58.381482 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381325    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f59179dd-36c5-404f-837f-8e54b1c799ed-config-volume\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:58.381482 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381362    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-ca-trust-extracted\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.381482 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6-signing-key\") pod \"service-ca-58598695f7-gwdvz\" (UID: \"3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6\") " pod="openshift-service-ca/service-ca-58598695f7-gwdvz"
Jul 16 20:00:58.381482 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-trusted-ca\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.381482 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.381791 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.381500    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:00:58.381791 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381543    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-certificates\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.381791 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.381555    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls podName:13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1 nodeName:}" failed. No retries permitted until 2026-07-16 20:00:58.881540354 +0000 UTC m=+37.132161091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4lrz" (UID: "13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:00:58.381791 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381569    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6-signing-cabundle\") pod \"service-ca-58598695f7-gwdvz\" (UID: \"3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6\") " pod="openshift-service-ca/service-ca-58598695f7-gwdvz"
Jul 16 20:00:58.381791 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.381791 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.381771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5jh7\" (UniqueName: \"kubernetes.io/projected/bb00e945-1b1b-4387-8834-cd0365bdcd23-kube-api-access-k5jh7\") pod \"ingress-canary-l4947\" (UID: \"bb00e945-1b1b-4387-8834-cd0365bdcd23\") " pod="openshift-ingress-canary/ingress-canary-l4947"
Jul 16 20:00:58.382132 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.381838    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:00:58.382132 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.381850    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff7887449-bq987: secret "image-registry-tls" not found
Jul 16 20:00:58.382132 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.381894    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls podName:aa58ca0b-1f88-4f88-a06c-a93efba1fd37 nodeName:}" failed. No retries permitted until 2026-07-16 20:00:58.881877452 +0000 UTC m=+37.132498218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls") pod "image-registry-ff7887449-bq987" (UID: "aa58ca0b-1f88-4f88-a06c-a93efba1fd37") : secret "image-registry-tls" not found
Jul 16 20:00:58.382403 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.382380    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-trusted-ca\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.385376 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.385326    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-installation-pull-secrets\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.385458 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.385395    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6-signing-key\") pod \"service-ca-58598695f7-gwdvz\" (UID: \"3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6\") " pod="openshift-service-ca/service-ca-58598695f7-gwdvz"
Jul 16 20:00:58.385458 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.385402    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb00e945-1b1b-4387-8834-cd0365bdcd23-cert\") pod \"ingress-canary-l4947\" (UID: \"bb00e945-1b1b-4387-8834-cd0365bdcd23\") " pod="openshift-ingress-canary/ingress-canary-l4947"
Jul 16 20:00:58.385458 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.385411    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-image-registry-private-configuration\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.392443 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.392423    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-bound-sa-token\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.393086 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.393041    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5jh7\" (UniqueName: \"kubernetes.io/projected/bb00e945-1b1b-4387-8834-cd0365bdcd23-kube-api-access-k5jh7\") pod \"ingress-canary-l4947\" (UID: \"bb00e945-1b1b-4387-8834-cd0365bdcd23\") " pod="openshift-ingress-canary/ingress-canary-l4947"
Jul 16 20:00:58.393086 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.393056    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nh5x\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-kube-api-access-7nh5x\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.393227 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.393186    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbt4b\" (UniqueName: \"kubernetes.io/projected/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-api-access-lbt4b\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.393501 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.393484    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrrmf\" (UniqueName: \"kubernetes.io/projected/3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6-kube-api-access-rrrmf\") pod \"service-ca-58598695f7-gwdvz\" (UID: \"3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6\") " pod="openshift-service-ca/service-ca-58598695f7-gwdvz"
Jul 16 20:00:58.482771 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.482739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:58.482955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.482778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xccl5\" (UniqueName: \"kubernetes.io/projected/f59179dd-36c5-404f-837f-8e54b1c799ed-kube-api-access-xccl5\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:58.482955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.482795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f59179dd-36c5-404f-837f-8e54b1c799ed-config-volume\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:58.482955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.482882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59179dd-36c5-404f-837f-8e54b1c799ed-tmp-dir\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:58.482955 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.482892    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:00:58.482955 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.482957    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls podName:f59179dd-36c5-404f-837f-8e54b1c799ed nodeName:}" failed. No retries permitted until 2026-07-16 20:00:58.98294091 +0000 UTC m=+37.233561650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls") pod "dns-default-l4b9r" (UID: "f59179dd-36c5-404f-837f-8e54b1c799ed") : secret "dns-default-metrics-tls" not found
Jul 16 20:00:58.483222 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.483207    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59179dd-36c5-404f-837f-8e54b1c799ed-tmp-dir\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:58.483386 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.483369    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f59179dd-36c5-404f-837f-8e54b1c799ed-config-volume\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:58.492224 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.492200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xccl5\" (UniqueName: \"kubernetes.io/projected/f59179dd-36c5-404f-837f-8e54b1c799ed-kube-api-access-xccl5\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:58.496113 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.496100    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l4947"
Jul 16 20:00:58.501594 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.501571    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-gwdvz"
Jul 16 20:00:58.536661 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.536614    2576 generic.go:356] "Generic (PLEG): container finished" podID="29d77657-97fd-4e5a-97d9-63dea83fadf9" containerID="6daba157aa363eab945ace03986d60e1098de04929604e27704b0e43f56275b9" exitCode=0
Jul 16 20:00:58.536851 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.536669    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rwcb" event={"ID":"29d77657-97fd-4e5a-97d9-63dea83fadf9","Type":"ContainerDied","Data":"6daba157aa363eab945ace03986d60e1098de04929604e27704b0e43f56275b9"}
Jul 16 20:00:58.693551 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.693320    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l4947"]
Jul 16 20:00:58.694116 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.694093    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-gwdvz"]
Jul 16 20:00:58.695495 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:58.695465    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb00e945_1b1b_4387_8834_cd0365bdcd23.slice/crio-b5dd14ed6b82b492d123002dca1f41b0abf95446f7e115255ea95065b68dc15f WatchSource:0}: Error finding container b5dd14ed6b82b492d123002dca1f41b0abf95446f7e115255ea95065b68dc15f: Status 404 returned error can't find the container with id b5dd14ed6b82b492d123002dca1f41b0abf95446f7e115255ea95065b68dc15f
Jul 16 20:00:58.695991 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:00:58.695972    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acd03f5_4ae7_4dd4_97cc_9c74a3c5bbb6.slice/crio-0385738386ea5ab4906741efa8bdc9a7cb4a6d47a2d8a91f7e580fcced2e6867 WatchSource:0}: Error finding container 0385738386ea5ab4906741efa8bdc9a7cb4a6d47a2d8a91f7e580fcced2e6867: Status 404 returned error can't find the container with id 0385738386ea5ab4906741efa8bdc9a7cb4a6d47a2d8a91f7e580fcced2e6867
Jul 16 20:00:58.717820 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.717800    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5h765_97cc28a3-e813-4407-bad7-8f53dec993f4/node-ca/0.log"
Jul 16 20:00:58.886803 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.886774    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.886978 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.886820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:58.886978 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.886883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:58.886978 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.886938    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:00:58.886978 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.886958    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:00:58.887095 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.886998    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls podName:13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1 nodeName:}" failed. No retries permitted until 2026-07-16 20:00:59.88698208 +0000 UTC m=+38.137602818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4lrz" (UID: "13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:00:58.887095 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.887015    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm podName:13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1 nodeName:}" failed. No retries permitted until 2026-07-16 20:00:59.887007212 +0000 UTC m=+38.137627953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4lrz" (UID: "13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1") : configmap "kube-rbac-proxy" not found
Jul 16 20:00:58.887095 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.887049    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:00:58.887095 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.887065    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff7887449-bq987: secret "image-registry-tls" not found
Jul 16 20:00:58.887220 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.887110    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls podName:aa58ca0b-1f88-4f88-a06c-a93efba1fd37 nodeName:}" failed. No retries permitted until 2026-07-16 20:00:59.887097887 +0000 UTC m=+38.137718624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls") pod "image-registry-ff7887449-bq987" (UID: "aa58ca0b-1f88-4f88-a06c-a93efba1fd37") : secret "image-registry-tls" not found
Jul 16 20:00:58.987465 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:58.987385    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:58.987623 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.987554    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:00:58.987687 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:58.987628    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls podName:f59179dd-36c5-404f-837f-8e54b1c799ed nodeName:}" failed. No retries permitted until 2026-07-16 20:00:59.987607887 +0000 UTC m=+38.238228626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls") pod "dns-default-l4b9r" (UID: "f59179dd-36c5-404f-837f-8e54b1c799ed") : secret "dns-default-metrics-tls" not found
Jul 16 20:00:59.540560 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:59.540523    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-gwdvz" event={"ID":"3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6","Type":"ContainerStarted","Data":"0385738386ea5ab4906741efa8bdc9a7cb4a6d47a2d8a91f7e580fcced2e6867"}
Jul 16 20:00:59.541757 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:59.541734    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l4947" event={"ID":"bb00e945-1b1b-4387-8834-cd0365bdcd23","Type":"ContainerStarted","Data":"b5dd14ed6b82b492d123002dca1f41b0abf95446f7e115255ea95065b68dc15f"}
Jul 16 20:00:59.895061 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:59.895027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:59.895061 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:59.895071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:00:59.895298 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:59.895174    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:00:59.895298 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:59.895174    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:00:59.895298 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:59.895205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:00:59.895298 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:59.895235    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls podName:13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1 nodeName:}" failed. No retries permitted until 2026-07-16 20:01:01.895220215 +0000 UTC m=+40.145840952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4lrz" (UID: "13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:00:59.895298 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:59.895247    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:00:59.895298 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:59.895183    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff7887449-bq987: secret "image-registry-tls" not found
Jul 16 20:00:59.895298 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:59.895282    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm podName:13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1 nodeName:}" failed. No retries permitted until 2026-07-16 20:01:01.895271435 +0000 UTC m=+40.145892173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4lrz" (UID: "13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1") : configmap "kube-rbac-proxy" not found
Jul 16 20:00:59.895298 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:59.895293    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls podName:aa58ca0b-1f88-4f88-a06c-a93efba1fd37 nodeName:}" failed. No retries permitted until 2026-07-16 20:01:01.895286948 +0000 UTC m=+40.145907684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls") pod "image-registry-ff7887449-bq987" (UID: "aa58ca0b-1f88-4f88-a06c-a93efba1fd37") : secret "image-registry-tls" not found
Jul 16 20:00:59.996068 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:00:59.996040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:00:59.996252 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:59.996219    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:00:59.996392 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:00:59.996301    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls podName:f59179dd-36c5-404f-837f-8e54b1c799ed nodeName:}" failed. No retries permitted until 2026-07-16 20:01:01.996282894 +0000 UTC m=+40.246903639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls") pod "dns-default-l4b9r" (UID: "f59179dd-36c5-404f-837f-8e54b1c799ed") : secret "dns-default-metrics-tls" not found
Jul 16 20:01:00.548053 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:00.548014    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-gwdvz" event={"ID":"3acd03f5-4ae7-4dd4-97cc-9c74a3c5bbb6","Type":"ContainerStarted","Data":"bc02fa20ff8286f407e5c1a052bcbb6bc69117156e40669f6c4618f3925e07ec"}
Jul 16 20:01:01.915191 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:01.914734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:01:01.915191 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:01.914808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:01:01.915191 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:01.914884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:01:01.915191 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:01.915036    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:01:01.915191 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:01.915080    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:01:01.915191 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:01.915107    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls podName:13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1 nodeName:}" failed. No retries permitted until 2026-07-16 20:01:05.915085839 +0000 UTC m=+44.165706596 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4lrz" (UID: "13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:01:01.915191 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:01.915124    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm podName:13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1 nodeName:}" failed. No retries permitted until 2026-07-16 20:01:05.915114371 +0000 UTC m=+44.165735109 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4lrz" (UID: "13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1") : configmap "kube-rbac-proxy" not found
Jul 16 20:01:01.915191 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:01.915040    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:01:01.915191 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:01.915138    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff7887449-bq987: secret "image-registry-tls" not found
Jul 16 20:01:01.915191 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:01.915163    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls podName:aa58ca0b-1f88-4f88-a06c-a93efba1fd37 nodeName:}" failed. No retries permitted until 2026-07-16 20:01:05.915155538 +0000 UTC m=+44.165776289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls") pod "image-registry-ff7887449-bq987" (UID: "aa58ca0b-1f88-4f88-a06c-a93efba1fd37") : secret "image-registry-tls" not found
Jul 16 20:01:02.016773 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:02.016140    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:01:02.016773 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:02.016352    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:01:02.016773 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:02.016414    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls podName:f59179dd-36c5-404f-837f-8e54b1c799ed nodeName:}" failed. No retries permitted until 2026-07-16 20:01:06.016394466 +0000 UTC m=+44.267015256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls") pod "dns-default-l4b9r" (UID: "f59179dd-36c5-404f-837f-8e54b1c799ed") : secret "dns-default-metrics-tls" not found
Jul 16 20:01:03.557399 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:03.557365    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l4947" event={"ID":"bb00e945-1b1b-4387-8834-cd0365bdcd23","Type":"ContainerStarted","Data":"0479c424f48cc4d5f1e1594a0e885672e59a772599156484f9f56898c30007bb"}
Jul 16 20:01:03.578483 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:03.578435    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l4947" podStartSLOduration=1.453029279 podStartE2EDuration="5.578414809s" totalImagesPullingTime="4.12538553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:03.577032251 +0000 UTC m=+41.827653014" watchObservedRunningTime="2026-07-16 20:01:03.578414809 +0000 UTC m=+41.829035570"
Jul 16 20:01:03.578862 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:03.578832    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-gwdvz" podStartSLOduration=3.871228372 podStartE2EDuration="5.578822273s" totalImagesPullingTime="1.707593901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:00.566030356 +0000 UTC m=+38.816651115" watchObservedRunningTime="2026-07-16 20:01:03.578822273 +0000 UTC m=+41.829443009"
Jul 16 20:01:05.953624 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:05.953583    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:01:05.954101 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:05.953683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:01:05.954101 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:05.953781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:01:05.954101 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:05.953898    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:01:05.954101 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:05.953960    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls podName:13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1 nodeName:}" failed. No retries permitted until 2026-07-16 20:01:13.953942997 +0000 UTC m=+52.204563737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4lrz" (UID: "13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:01:05.954101 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:05.954020    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:01:05.954101 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:05.954046    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm podName:13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1 nodeName:}" failed. No retries permitted until 2026-07-16 20:01:13.954038446 +0000 UTC m=+52.204659183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4lrz" (UID: "13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1") : configmap "kube-rbac-proxy" not found
Jul 16 20:01:05.958642 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:05.958615    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls\") pod \"image-registry-ff7887449-bq987\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") " pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:01:05.983974 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:05.983945    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:01:06.054655 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:06.054612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:01:06.057488 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:06.057466    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59179dd-36c5-404f-837f-8e54b1c799ed-metrics-tls\") pod \"dns-default-l4b9r\" (UID: \"f59179dd-36c5-404f-837f-8e54b1c799ed\") " pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:01:06.318608 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:06.318368    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:01:07.131534 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:07.131500    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l4b9r"]
Jul 16 20:01:07.134644 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:01:07.134618    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59179dd_36c5_404f_837f_8e54b1c799ed.slice/crio-5bc1cfc2a0b781c30b01034b07f13612066155e357f7b552c5aede3d830ee2c9 WatchSource:0}: Error finding container 5bc1cfc2a0b781c30b01034b07f13612066155e357f7b552c5aede3d830ee2c9: Status 404 returned error can't find the container with id 5bc1cfc2a0b781c30b01034b07f13612066155e357f7b552c5aede3d830ee2c9
Jul 16 20:01:07.148897 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:07.148875    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ff7887449-bq987"]
Jul 16 20:01:07.304990 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:01:07.304905    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa58ca0b_1f88_4f88_a06c_a93efba1fd37.slice/crio-dc348d9b9ee2a58cf5434ca8e4d9ccbab7b32561c62548748ae87022acb3649f WatchSource:0}: Error finding container dc348d9b9ee2a58cf5434ca8e4d9ccbab7b32561c62548748ae87022acb3649f: Status 404 returned error can't find the container with id dc348d9b9ee2a58cf5434ca8e4d9ccbab7b32561c62548748ae87022acb3649f
Jul 16 20:01:07.570273 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:07.570243    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rwcb" event={"ID":"29d77657-97fd-4e5a-97d9-63dea83fadf9","Type":"ContainerStarted","Data":"8271e21f20e5ff9edf4c9b0719faa52a47c87650b6a2f8e14bef6d7281394bad"}
Jul 16 20:01:07.572291 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:07.572261    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ff7887449-bq987" event={"ID":"aa58ca0b-1f88-4f88-a06c-a93efba1fd37","Type":"ContainerStarted","Data":"941d9c46e85baf3367574fd717e4e3dcdc8ad25d54f9e8dd9b67c65475cbc567"}
Jul 16 20:01:07.572394 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:07.572315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ff7887449-bq987" event={"ID":"aa58ca0b-1f88-4f88-a06c-a93efba1fd37","Type":"ContainerStarted","Data":"dc348d9b9ee2a58cf5434ca8e4d9ccbab7b32561c62548748ae87022acb3649f"}
Jul 16 20:01:07.572602 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:07.572515    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:01:07.573544 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:07.573506    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l4b9r" event={"ID":"f59179dd-36c5-404f-837f-8e54b1c799ed","Type":"ContainerStarted","Data":"5bc1cfc2a0b781c30b01034b07f13612066155e357f7b552c5aede3d830ee2c9"}
Jul 16 20:01:07.612140 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:07.612084    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-ff7887449-bq987" podStartSLOduration=40.61206167 podStartE2EDuration="40.61206167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:07.611613398 +0000 UTC m=+45.862234161" watchObservedRunningTime="2026-07-16 20:01:07.61206167 +0000 UTC m=+45.862682430"
Jul 16 20:01:08.579355 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:08.579316    2576 generic.go:356] "Generic (PLEG): container finished" podID="29d77657-97fd-4e5a-97d9-63dea83fadf9" containerID="8271e21f20e5ff9edf4c9b0719faa52a47c87650b6a2f8e14bef6d7281394bad" exitCode=0
Jul 16 20:01:08.579897 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:08.579366    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rwcb" event={"ID":"29d77657-97fd-4e5a-97d9-63dea83fadf9","Type":"ContainerDied","Data":"8271e21f20e5ff9edf4c9b0719faa52a47c87650b6a2f8e14bef6d7281394bad"}
Jul 16 20:01:09.585090 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:09.585054    2576 generic.go:356] "Generic (PLEG): container finished" podID="29d77657-97fd-4e5a-97d9-63dea83fadf9" containerID="098a19e80aa6a21356d21a5cf6f7cf3e64557ce487a04811194843b8a7de5a07" exitCode=0
Jul 16 20:01:09.585522 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:09.585133    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rwcb" event={"ID":"29d77657-97fd-4e5a-97d9-63dea83fadf9","Type":"ContainerDied","Data":"098a19e80aa6a21356d21a5cf6f7cf3e64557ce487a04811194843b8a7de5a07"}
Jul 16 20:01:09.587011 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:09.586991    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l4b9r" event={"ID":"f59179dd-36c5-404f-837f-8e54b1c799ed","Type":"ContainerStarted","Data":"cd9555bc33e27cd8bfd431ccdeb3d99e109b21f75d0de4a37ef6fb4c63aca134"}
Jul 16 20:01:09.587109 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:09.587037    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l4b9r" event={"ID":"f59179dd-36c5-404f-837f-8e54b1c799ed","Type":"ContainerStarted","Data":"cb0dd63fbe27189024d4da5f05b7b30ab475eca4dd652e7160b458adf7c60f8b"}
Jul 16 20:01:09.587202 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:09.587189    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:01:09.630355 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:09.630289    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l4b9r" podStartSLOduration=9.684324335 podStartE2EDuration="11.630268689s" totalImagesPullingTime="1.945944354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:09.629897616 +0000 UTC m=+47.880518376" watchObservedRunningTime="2026-07-16 20:01:09.630268689 +0000 UTC m=+47.880889449"
Jul 16 20:01:10.592842 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:10.592809    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rwcb" event={"ID":"29d77657-97fd-4e5a-97d9-63dea83fadf9","Type":"ContainerStarted","Data":"355d1ce61517ed93f7b71aea7f34e9c0809330afd91c8b37814e1ddb610648a6"}
Jul 16 20:01:10.618416 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:10.618366    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4rwcb" podStartSLOduration=7.91566952 podStartE2EDuration="47.618351479s" totalImagesPullingTime="39.702681959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:00:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:10.616900926 +0000 UTC m=+48.867521682" watchObservedRunningTime="2026-07-16 20:01:10.618351479 +0000 UTC m=+48.868972237"
Jul 16 20:01:14.031901 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:14.031860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:01:14.032355 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:14.031916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:01:14.041066 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:14.041039    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:01:14.041066 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:14.041060    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4lrz\" (UID: \"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1\") " pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:01:14.108137 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:14.108105    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4lrz"
Jul 16 20:01:14.250403 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:14.250372    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4lrz"]
Jul 16 20:01:14.253571 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:01:14.253529    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ef9ffe_f7cd_4baf_826b_c05b6b5f8ea1.slice/crio-461e4c92728b57c3b99dda5af8bced6ff121cbf40da9925f5a92c349eca6089c WatchSource:0}: Error finding container 461e4c92728b57c3b99dda5af8bced6ff121cbf40da9925f5a92c349eca6089c: Status 404 returned error can't find the container with id 461e4c92728b57c3b99dda5af8bced6ff121cbf40da9925f5a92c349eca6089c
Jul 16 20:01:14.604817 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:14.604781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4lrz" event={"ID":"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1","Type":"ContainerStarted","Data":"461e4c92728b57c3b99dda5af8bced6ff121cbf40da9925f5a92c349eca6089c"}
Jul 16 20:01:19.595368 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:19.595335    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l4b9r"
Jul 16 20:01:19.621481 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:19.621403    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4lrz" event={"ID":"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1","Type":"ContainerStarted","Data":"96408d205692131731c0289a87aea6a75ef2c7b75511ccfc750d86886151ddfb"}
Jul 16 20:01:22.330834 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.330769    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-m8ht5"]
Jul 16 20:01:22.335608 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.335584    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-z62x9"]
Jul 16 20:01:22.336473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.336454    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-m8ht5"
Jul 16 20:01:22.337513 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.337493    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-z62x9"
Jul 16 20:01:22.340622 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.339428    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 20:01:22.341535 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.341515    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 20:01:22.341641 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.341558    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5fq82\""
Jul 16 20:01:22.341743 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.341722    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 16 20:01:22.341836 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.341819    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-tgft6\""
Jul 16 20:01:22.349066 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.349046    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-ff7887449-bq987"]
Jul 16 20:01:22.354847 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.354780    2576 patch_prober.go:28] interesting pod/image-registry-ff7887449-bq987 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 16 20:01:22.354963 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.354886    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-ff7887449-bq987" podUID="aa58ca0b-1f88-4f88-a06c-a93efba1fd37" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 20:01:22.357574 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.357550    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-z62x9"]
Jul 16 20:01:22.368035 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.367792    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-m8ht5"]
Jul 16 20:01:22.402487 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.402463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwl7r\" (UniqueName: \"kubernetes.io/projected/10f346ba-a76a-45c8-bab6-3591fa1dcccd-kube-api-access-hwl7r\") pod \"downloads-5bc67dcd46-m8ht5\" (UID: \"10f346ba-a76a-45c8-bab6-3591fa1dcccd\") " pod="openshift-console/downloads-5bc67dcd46-m8ht5"
Jul 16 20:01:22.402613 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.402500    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4bad754-60b3-43ed-aae1-de6e5015c3ef-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-z62x9\" (UID: \"f4bad754-60b3-43ed-aae1-de6e5015c3ef\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-z62x9"
Jul 16 20:01:22.441283 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.441248    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f46679889-f9szw"]
Jul 16 20:01:22.445796 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.445765    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.463045 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.463021    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f46679889-f9szw"]
Jul 16 20:01:22.503789 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.503759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2475315-9baa-456c-b86a-c97a0f1a04a4-trusted-ca\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.503925 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.503815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2475315-9baa-456c-b86a-c97a0f1a04a4-ca-trust-extracted\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.503925 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.503884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4bad754-60b3-43ed-aae1-de6e5015c3ef-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-z62x9\" (UID: \"f4bad754-60b3-43ed-aae1-de6e5015c3ef\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-z62x9"
Jul 16 20:01:22.504054 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.503946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2475315-9baa-456c-b86a-c97a0f1a04a4-installation-pull-secrets\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.504054 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.504017    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2475315-9baa-456c-b86a-c97a0f1a04a4-image-registry-private-configuration\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.504054 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.504050    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2475315-9baa-456c-b86a-c97a0f1a04a4-bound-sa-token\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.504160 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.504085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2475315-9baa-456c-b86a-c97a0f1a04a4-registry-tls\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.504160 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.504101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2475315-9baa-456c-b86a-c97a0f1a04a4-registry-certificates\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.504160 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.504120    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwl7r\" (UniqueName: \"kubernetes.io/projected/10f346ba-a76a-45c8-bab6-3591fa1dcccd-kube-api-access-hwl7r\") pod \"downloads-5bc67dcd46-m8ht5\" (UID: \"10f346ba-a76a-45c8-bab6-3591fa1dcccd\") " pod="openshift-console/downloads-5bc67dcd46-m8ht5"
Jul 16 20:01:22.504160 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.504136    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x2xx\" (UniqueName: \"kubernetes.io/projected/b2475315-9baa-456c-b86a-c97a0f1a04a4-kube-api-access-7x2xx\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.506607 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.506584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f4bad754-60b3-43ed-aae1-de6e5015c3ef-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-z62x9\" (UID: \"f4bad754-60b3-43ed-aae1-de6e5015c3ef\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-z62x9"
Jul 16 20:01:22.513926 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.513900    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwl7r\" (UniqueName: \"kubernetes.io/projected/10f346ba-a76a-45c8-bab6-3591fa1dcccd-kube-api-access-hwl7r\") pod \"downloads-5bc67dcd46-m8ht5\" (UID: \"10f346ba-a76a-45c8-bab6-3591fa1dcccd\") " pod="openshift-console/downloads-5bc67dcd46-m8ht5"
Jul 16 20:01:22.605386 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.605303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2475315-9baa-456c-b86a-c97a0f1a04a4-trusted-ca\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.605386 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.605371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2475315-9baa-456c-b86a-c97a0f1a04a4-ca-trust-extracted\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.605608 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.605426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2475315-9baa-456c-b86a-c97a0f1a04a4-installation-pull-secrets\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.605608 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.605484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2475315-9baa-456c-b86a-c97a0f1a04a4-image-registry-private-configuration\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.605608 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.605529    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2475315-9baa-456c-b86a-c97a0f1a04a4-bound-sa-token\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.605608 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.605569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2475315-9baa-456c-b86a-c97a0f1a04a4-registry-tls\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.605608 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.605595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2475315-9baa-456c-b86a-c97a0f1a04a4-registry-certificates\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.605877 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.605626    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x2xx\" (UniqueName: \"kubernetes.io/projected/b2475315-9baa-456c-b86a-c97a0f1a04a4-kube-api-access-7x2xx\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.605877 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.605815    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2475315-9baa-456c-b86a-c97a0f1a04a4-ca-trust-extracted\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.607197 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.607171    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2475315-9baa-456c-b86a-c97a0f1a04a4-registry-certificates\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.607665 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.607627    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2475315-9baa-456c-b86a-c97a0f1a04a4-trusted-ca\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.608879 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.608854    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2475315-9baa-456c-b86a-c97a0f1a04a4-installation-pull-secrets\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.608972 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.608903    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2475315-9baa-456c-b86a-c97a0f1a04a4-registry-tls\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.609314 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.609293    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2475315-9baa-456c-b86a-c97a0f1a04a4-image-registry-private-configuration\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.617485 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.617431    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x2xx\" (UniqueName: \"kubernetes.io/projected/b2475315-9baa-456c-b86a-c97a0f1a04a4-kube-api-access-7x2xx\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.620955 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.620936    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2475315-9baa-456c-b86a-c97a0f1a04a4-bound-sa-token\") pod \"image-registry-f46679889-f9szw\" (UID: \"b2475315-9baa-456c-b86a-c97a0f1a04a4\") " pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:22.651647 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.651623    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-m8ht5"
Jul 16 20:01:22.659775 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.659754    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-z62x9"
Jul 16 20:01:22.756039 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:22.756013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:23.002872 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:23.002832    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-z62x9"]
Jul 16 20:01:23.004667 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:01:23.004593    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bad754_60b3_43ed_aae1_de6e5015c3ef.slice/crio-4464b68b42a68198c938651e1c98e2eaa6466f8605b08b9c3f59db3b13601b8d WatchSource:0}: Error finding container 4464b68b42a68198c938651e1c98e2eaa6466f8605b08b9c3f59db3b13601b8d: Status 404 returned error can't find the container with id 4464b68b42a68198c938651e1c98e2eaa6466f8605b08b9c3f59db3b13601b8d
Jul 16 20:01:23.037524 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:23.037501    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-m8ht5"]
Jul 16 20:01:23.039771 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:01:23.039749    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f346ba_a76a_45c8_bab6_3591fa1dcccd.slice/crio-06d5bfb66245ca912a8305f6e43f247126ad9981c6de5a750e2959ff2f1e6379 WatchSource:0}: Error finding container 06d5bfb66245ca912a8305f6e43f247126ad9981c6de5a750e2959ff2f1e6379: Status 404 returned error can't find the container with id 06d5bfb66245ca912a8305f6e43f247126ad9981c6de5a750e2959ff2f1e6379
Jul 16 20:01:23.043139 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:23.043110    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f46679889-f9szw"]
Jul 16 20:01:23.045880 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:01:23.045861    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2475315_9baa_456c_b86a_c97a0f1a04a4.slice/crio-e278122448b4c55d7ac6291571b4ab559720f549b042449cf889b2263903e08c WatchSource:0}: Error finding container e278122448b4c55d7ac6291571b4ab559720f549b042449cf889b2263903e08c: Status 404 returned error can't find the container with id e278122448b4c55d7ac6291571b4ab559720f549b042449cf889b2263903e08c
Jul 16 20:01:23.637455 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:23.637418    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f46679889-f9szw" event={"ID":"b2475315-9baa-456c-b86a-c97a0f1a04a4","Type":"ContainerStarted","Data":"63858d5162b1466452c395e795b9dbeb2115ca1d41934e16d6c13d09832d498a"}
Jul 16 20:01:23.637911 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:23.637477    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f46679889-f9szw" event={"ID":"b2475315-9baa-456c-b86a-c97a0f1a04a4","Type":"ContainerStarted","Data":"e278122448b4c55d7ac6291571b4ab559720f549b042449cf889b2263903e08c"}
Jul 16 20:01:23.637911 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:23.637514    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:23.639552 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:23.639521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4lrz" event={"ID":"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1","Type":"ContainerStarted","Data":"705c9fedbf6ff8de26b6a48ddbf34d833eb2a833e9e7321ffab9d4a76abcb659"}
Jul 16 20:01:23.640895 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:23.640864    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-m8ht5" event={"ID":"10f346ba-a76a-45c8-bab6-3591fa1dcccd","Type":"ContainerStarted","Data":"06d5bfb66245ca912a8305f6e43f247126ad9981c6de5a750e2959ff2f1e6379"}
Jul 16 20:01:23.642241 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:23.642214    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-z62x9" event={"ID":"f4bad754-60b3-43ed-aae1-de6e5015c3ef","Type":"ContainerStarted","Data":"4464b68b42a68198c938651e1c98e2eaa6466f8605b08b9c3f59db3b13601b8d"}
Jul 16 20:01:23.661100 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:23.661057    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f46679889-f9szw" podStartSLOduration=1.661044105 podStartE2EDuration="1.661044105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:23.660944332 +0000 UTC m=+61.911565095" watchObservedRunningTime="2026-07-16 20:01:23.661044105 +0000 UTC m=+61.911664855"
Jul 16 20:01:24.541006 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.540986    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srq56"
Jul 16 20:01:24.647760 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.647693    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-z62x9" event={"ID":"f4bad754-60b3-43ed-aae1-de6e5015c3ef","Type":"ContainerStarted","Data":"061d27d3aa499a6dfe05ab87868fa06719b301e1035c5efc71a63935e605cc31"}
Jul 16 20:01:24.748824 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.748789    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d7fd8d5df-4xzd8"]
Jul 16 20:01:24.768832 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.768808    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d7fd8d5df-4xzd8"]
Jul 16 20:01:24.769001 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.768907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.771443 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.771421    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 16 20:01:24.771443 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.771422    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 16 20:01:24.771651 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.771422    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 16 20:01:24.772231 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.771818    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 16 20:01:24.772231 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.771938    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 16 20:01:24.772231 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.772061    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z6795\""
Jul 16 20:01:24.826085 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.826052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/904f4feb-d288-4671-a63b-9109773aaf56-console-oauth-config\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.826085 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.826089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-oauth-serving-cert\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.826300 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.826113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hldv\" (UniqueName: \"kubernetes.io/projected/904f4feb-d288-4671-a63b-9109773aaf56-kube-api-access-2hldv\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.826300 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.826149    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/904f4feb-d288-4671-a63b-9109773aaf56-console-serving-cert\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.826300 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.826202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-service-ca\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.826300 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.826288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-console-config\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.927440 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.927392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-console-config\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.927603 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.927476    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/904f4feb-d288-4671-a63b-9109773aaf56-console-oauth-config\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.927603 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.927502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-oauth-serving-cert\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.927603 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.927529    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hldv\" (UniqueName: \"kubernetes.io/projected/904f4feb-d288-4671-a63b-9109773aaf56-kube-api-access-2hldv\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.927603 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.927563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/904f4feb-d288-4671-a63b-9109773aaf56-console-serving-cert\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.927603 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.927595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-service-ca\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.928295 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.928263    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-service-ca\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.928395 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.928267    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-console-config\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.928395 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.928325    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-oauth-serving-cert\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.930855 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.930832    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/904f4feb-d288-4671-a63b-9109773aaf56-console-serving-cert\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.931153 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.931133    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/904f4feb-d288-4671-a63b-9109773aaf56-console-oauth-config\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:24.942164 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:24.942132    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hldv\" (UniqueName: \"kubernetes.io/projected/904f4feb-d288-4671-a63b-9109773aaf56-kube-api-access-2hldv\") pod \"console-6d7fd8d5df-4xzd8\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") " pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:25.079825 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:25.079737    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:25.236589 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:25.236549    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d7fd8d5df-4xzd8"]
Jul 16 20:01:25.239665 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:01:25.239636    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904f4feb_d288_4671_a63b_9109773aaf56.slice/crio-d6ab2bd39367a44b7cd8a0bc9bc3222d9feb7f2633bceb9aea86863cdbb5f03d WatchSource:0}: Error finding container d6ab2bd39367a44b7cd8a0bc9bc3222d9feb7f2633bceb9aea86863cdbb5f03d: Status 404 returned error can't find the container with id d6ab2bd39367a44b7cd8a0bc9bc3222d9feb7f2633bceb9aea86863cdbb5f03d
Jul 16 20:01:25.652542 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:25.652502    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7fd8d5df-4xzd8" event={"ID":"904f4feb-d288-4671-a63b-9109773aaf56","Type":"ContainerStarted","Data":"d6ab2bd39367a44b7cd8a0bc9bc3222d9feb7f2633bceb9aea86863cdbb5f03d"}
Jul 16 20:01:25.653026 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:25.652734    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-z62x9"
Jul 16 20:01:25.659409 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:25.659386    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-z62x9"
Jul 16 20:01:25.670545 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:25.670491    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-z62x9" podStartSLOduration=2.146528568 podStartE2EDuration="3.670472352s" totalImagesPullingTime="1.523943784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:25.669235426 +0000 UTC m=+63.919856212" watchObservedRunningTime="2026-07-16 20:01:25.670472352 +0000 UTC m=+63.921093111"
Jul 16 20:01:26.742191 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.741132    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"]
Jul 16 20:01:26.744092 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.743416    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:26.746264 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.746239    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 20:01:26.746374 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.746297    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 20:01:26.746502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.746487    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 16 20:01:26.747506 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.747486    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8vd9l\""
Jul 16 20:01:26.747713 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.747683    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 20:01:26.747912 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.747896    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 16 20:01:26.753022 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.752999    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"]
Jul 16 20:01:26.846858 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.846820    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-78rw2\" (UID: \"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:26.847035 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.846900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-78rw2\" (UID: \"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:26.847035 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.846929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wgd\" (UniqueName: \"kubernetes.io/projected/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-kube-api-access-67wgd\") pod \"prometheus-operator-5945b5cfd-78rw2\" (UID: \"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:26.847035 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.846972    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-78rw2\" (UID: \"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:26.948233 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.948138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-78rw2\" (UID: \"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:26.948400 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.948239    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-78rw2\" (UID: \"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:26.948400 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.948269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wgd\" (UniqueName: \"kubernetes.io/projected/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-kube-api-access-67wgd\") pod \"prometheus-operator-5945b5cfd-78rw2\" (UID: \"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:26.948400 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.948332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-78rw2\" (UID: \"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:26.948566 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:26.948443    2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 16 20:01:26.948566 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:26.948506    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-prometheus-operator-tls podName:8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe nodeName:}" failed. No retries permitted until 2026-07-16 20:01:27.448488374 +0000 UTC m=+65.699109115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-78rw2" (UID: "8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe") : secret "prometheus-operator-tls" not found
Jul 16 20:01:26.949039 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.949012    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-78rw2\" (UID: \"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:26.958562 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.958506    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-78rw2\" (UID: \"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:26.960667 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:26.960643    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wgd\" (UniqueName: \"kubernetes.io/projected/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-kube-api-access-67wgd\") pod \"prometheus-operator-5945b5cfd-78rw2\" (UID: \"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:27.453447 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:27.453411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-78rw2\" (UID: \"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:27.458118 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:27.458090    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-78rw2\" (UID: \"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:27.656330 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:27.656294    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"
Jul 16 20:01:28.059405 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.058955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:01:28.059405 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.059016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs\") pod \"network-metrics-daemon-6ffx6\" (UID: \"0238483d-389c-44cf-92b1-7f8cf040159e\") " pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:01:28.061587 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.061559    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 20:01:28.061758 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.061658    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 20:01:28.072424 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.072399    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9711524-15cd-4165-854a-1453f37e4abc-original-pull-secret\") pod \"global-pull-secret-syncer-szlh8\" (UID: \"f9711524-15cd-4165-854a-1453f37e4abc\") " pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:01:28.072557 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.072469    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0238483d-389c-44cf-92b1-7f8cf040159e-metrics-certs\") pod \"network-metrics-daemon-6ffx6\" (UID: \"0238483d-389c-44cf-92b1-7f8cf040159e\") " pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:01:28.160126 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.160087    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpskv\" (UniqueName: \"kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv\") pod \"network-check-target-7jgvl\" (UID: \"ac9ea87f-36ee-4233-8037-2e3802ae7ae6\") " pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:01:28.163128 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.162923    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 20:01:28.173510 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.173482    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 20:01:28.184632 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.184604    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpskv\" (UniqueName: \"kubernetes.io/projected/ac9ea87f-36ee-4233-8037-2e3802ae7ae6-kube-api-access-rpskv\") pod \"network-check-target-7jgvl\" (UID: \"ac9ea87f-36ee-4233-8037-2e3802ae7ae6\") " pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:01:28.255622 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.255591    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nxdf6\""
Jul 16 20:01:28.260489 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.260300    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfzlm\""
Jul 16 20:01:28.263463 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.263342    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:01:28.263463 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.263401    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlh8"
Jul 16 20:01:28.268624 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.268601    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6ffx6"
Jul 16 20:01:28.538661 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.538568    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6ffx6"]
Jul 16 20:01:28.549565 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:01:28.549534    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0238483d_389c_44cf_92b1_7f8cf040159e.slice/crio-4351f70fd290279694ebc8d9e51f0c84125ec68c1b6d8fe167e036a859886556 WatchSource:0}: Error finding container 4351f70fd290279694ebc8d9e51f0c84125ec68c1b6d8fe167e036a859886556: Status 404 returned error can't find the container with id 4351f70fd290279694ebc8d9e51f0c84125ec68c1b6d8fe167e036a859886556
Jul 16 20:01:28.566222 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.566192    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7jgvl"]
Jul 16 20:01:28.588423 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.588394    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-szlh8"]
Jul 16 20:01:28.591447 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:01:28.591422    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9711524_15cd_4165_854a_1453f37e4abc.slice/crio-bdb81bd3d10838764e3c98af14fcc0ad8268e64a2ef2d083bbdeb9c82acb63d8 WatchSource:0}: Error finding container bdb81bd3d10838764e3c98af14fcc0ad8268e64a2ef2d083bbdeb9c82acb63d8: Status 404 returned error can't find the container with id bdb81bd3d10838764e3c98af14fcc0ad8268e64a2ef2d083bbdeb9c82acb63d8
Jul 16 20:01:28.666031 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.665946    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7fd8d5df-4xzd8" event={"ID":"904f4feb-d288-4671-a63b-9109773aaf56","Type":"ContainerStarted","Data":"8a33c46886ac63e56902acc28b44f5afdbdd4cd36308387e23bb034e398b3ff9"}
Jul 16 20:01:28.667314 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.667289    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7jgvl" event={"ID":"ac9ea87f-36ee-4233-8037-2e3802ae7ae6","Type":"ContainerStarted","Data":"6901de4390fe145502059c3f94cd40225ac8ef4498c5d51791cfe82b39d9e777"}
Jul 16 20:01:28.668555 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.668533    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6ffx6" event={"ID":"0238483d-389c-44cf-92b1-7f8cf040159e","Type":"ContainerStarted","Data":"4351f70fd290279694ebc8d9e51f0c84125ec68c1b6d8fe167e036a859886556"}
Jul 16 20:01:28.669977 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.669943    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-szlh8" event={"ID":"f9711524-15cd-4165-854a-1453f37e4abc","Type":"ContainerStarted","Data":"bdb81bd3d10838764e3c98af14fcc0ad8268e64a2ef2d083bbdeb9c82acb63d8"}
Jul 16 20:01:28.687020 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.686972    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d7fd8d5df-4xzd8" podStartSLOduration=1.544915617 podStartE2EDuration="4.686954914s" totalImagesPullingTime="3.142039297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:01:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:28.686517994 +0000 UTC m=+66.937138754" watchObservedRunningTime="2026-07-16 20:01:28.686954914 +0000 UTC m=+66.937575651"
Jul 16 20:01:28.800099 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:28.800069    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-78rw2"]
Jul 16 20:01:28.801496 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:01:28.801469    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed5ac0b_45e3_433e_8d0d_a7809b55fbfe.slice/crio-e192372b5cf249b04e7b8667309f05a0e6ee405ea09e4a4bf105cc0090fb8408 WatchSource:0}: Error finding container e192372b5cf249b04e7b8667309f05a0e6ee405ea09e4a4bf105cc0090fb8408: Status 404 returned error can't find the container with id e192372b5cf249b04e7b8667309f05a0e6ee405ea09e4a4bf105cc0090fb8408
Jul 16 20:01:29.677806 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:29.677766    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2" event={"ID":"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe","Type":"ContainerStarted","Data":"e192372b5cf249b04e7b8667309f05a0e6ee405ea09e4a4bf105cc0090fb8408"}
Jul 16 20:01:31.692592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:31.692556    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6ffx6" event={"ID":"0238483d-389c-44cf-92b1-7f8cf040159e","Type":"ContainerStarted","Data":"eac4480d7d63f4b0e4a8771704fba17310de5cf687374f3b4063695bcddac67f"}
Jul 16 20:01:31.693121 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:31.692611    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6ffx6" event={"ID":"0238483d-389c-44cf-92b1-7f8cf040159e","Type":"ContainerStarted","Data":"6d14ac36c801a9864a4610d052df7fb6c76e7743e0539515d0bd29bb9fb46ea8"}
Jul 16 20:01:31.711673 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:31.711602    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6ffx6" podStartSLOduration=66.463757284 podStartE2EDuration="1m8.711580051s" totalImagesPullingTime="2.247822767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:31.709950613 +0000 UTC m=+69.960571376" watchObservedRunningTime="2026-07-16 20:01:31.711580051 +0000 UTC m=+69.962200812"
Jul 16 20:01:32.355426 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:32.355391    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:01:35.080511 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:35.080471    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:35.083201 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:35.082765    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:35.088517 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:35.088490    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:35.712282 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:35.712253    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:01:42.766612 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:42.766493    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d7fd8d5df-4xzd8"]
Jul 16 20:01:44.653298 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:44.653057    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f46679889-f9szw"
Jul 16 20:01:44.742018 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:44.741986    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-szlh8" event={"ID":"f9711524-15cd-4165-854a-1453f37e4abc","Type":"ContainerStarted","Data":"fdfaebc581f8a80568c4f20aab4453e7e00d6704d7ae695792215e35afc24104"}
Jul 16 20:01:44.743814 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:44.743791    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4lrz" event={"ID":"13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1","Type":"ContainerStarted","Data":"644c06e4650bf26818a15a8d978b0549c94bfc2a9f169b5155fcda9f45f5160e"}
Jul 16 20:01:44.745357 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:44.745335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-m8ht5" event={"ID":"10f346ba-a76a-45c8-bab6-3591fa1dcccd","Type":"ContainerStarted","Data":"3c0b805f28828d4a7832bb908bb56c207c3bd069960fa2c917203de36b709b0a"}
Jul 16 20:01:44.745528 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:44.745509    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-m8ht5"
Jul 16 20:01:44.746995 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:44.746961    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7jgvl" event={"ID":"ac9ea87f-36ee-4233-8037-2e3802ae7ae6","Type":"ContainerStarted","Data":"14ca7a9013d1b7a6dcc44cae6453ca17463d28e7e3f4c2f14da840051f7777b2"}
Jul 16 20:01:44.747134 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:44.747107    2576 patch_prober.go:28] interesting pod/downloads-5bc67dcd46-m8ht5 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.11:8080/\": dial tcp 10.133.0.11:8080: connect: connection refused" start-of-body=
Jul 16 20:01:44.747134 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:44.747124    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:01:44.747252 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:44.747157    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5bc67dcd46-m8ht5" podUID="10f346ba-a76a-45c8-bab6-3591fa1dcccd" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.11:8080/\": dial tcp 10.133.0.11:8080: connect: connection refused"
Jul 16 20:01:44.761829 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:44.761789    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-szlh8" podStartSLOduration=65.976461255 podStartE2EDuration="1m21.761773771s" totalImagesPullingTime="15.785312516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:44.761002414 +0000 UTC m=+83.011623176" watchObservedRunningTime="2026-07-16 20:01:44.761773771 +0000 UTC m=+83.012394529"
Jul 16 20:01:44.779385 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:44.779323    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s4lrz" podStartSLOduration=16.779557638 podStartE2EDuration="46.779305103s" totalImagesPullingTime="29.999747465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:00:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:44.778236969 +0000 UTC m=+83.028857753" watchObservedRunningTime="2026-07-16 20:01:44.779305103 +0000 UTC m=+83.029925863"
Jul 16 20:01:44.799090 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:44.799038    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-m8ht5" podStartSLOduration=1.39170846 podStartE2EDuration="22.799018134s" totalImagesPullingTime="21.407309674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:44.797825991 +0000 UTC m=+83.048446753" watchObservedRunningTime="2026-07-16 20:01:44.799018134 +0000 UTC m=+83.049638897"
Jul 16 20:01:44.818553 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:44.818462    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7jgvl" podStartSLOduration=66.011278384 podStartE2EDuration="1m21.818437988s" totalImagesPullingTime="15.807159604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:44.817787758 +0000 UTC m=+83.068408518" watchObservedRunningTime="2026-07-16 20:01:44.818437988 +0000 UTC m=+83.069058747"
Jul 16 20:01:45.753240 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:45.753195    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2" event={"ID":"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe","Type":"ContainerStarted","Data":"e26f99cd3efd9b1c8180823cdf6a8971c028e739bc1432069233dfa21b34e61e"}
Jul 16 20:01:45.753725 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:45.753273    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2" event={"ID":"8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe","Type":"ContainerStarted","Data":"566719a710540b08c3ce1fd292d7cd5fc661872dcf555942b52ef086220ea17d"}
Jul 16 20:01:45.763043 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:45.763011    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-m8ht5"
Jul 16 20:01:45.775139 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:45.775083    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-78rw2" podStartSLOduration=3.872846347 podStartE2EDuration="19.775062215s" totalImagesPullingTime="15.902215868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:45.773680501 +0000 UTC m=+84.024301284" watchObservedRunningTime="2026-07-16 20:01:45.775062215 +0000 UTC m=+84.025683004"
Jul 16 20:01:47.374430 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.374378    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-ff7887449-bq987" podUID="aa58ca0b-1f88-4f88-a06c-a93efba1fd37" containerName="registry" containerID="cri-o://941d9c46e85baf3367574fd717e4e3dcdc8ad25d54f9e8dd9b67c65475cbc567" gracePeriod=30
Jul 16 20:01:47.685459 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.685426    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:01:47.762035 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.761999    2576 generic.go:356] "Generic (PLEG): container finished" podID="aa58ca0b-1f88-4f88-a06c-a93efba1fd37" containerID="941d9c46e85baf3367574fd717e4e3dcdc8ad25d54f9e8dd9b67c65475cbc567" exitCode=0
Jul 16 20:01:47.762218 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.762060    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff7887449-bq987"
Jul 16 20:01:47.762218 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.762069    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ff7887449-bq987" event={"ID":"aa58ca0b-1f88-4f88-a06c-a93efba1fd37","Type":"ContainerDied","Data":"941d9c46e85baf3367574fd717e4e3dcdc8ad25d54f9e8dd9b67c65475cbc567"}
Jul 16 20:01:47.762218 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.762133    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ff7887449-bq987" event={"ID":"aa58ca0b-1f88-4f88-a06c-a93efba1fd37","Type":"ContainerDied","Data":"dc348d9b9ee2a58cf5434ca8e4d9ccbab7b32561c62548748ae87022acb3649f"}
Jul 16 20:01:47.762218 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.762177    2576 scope.go:117] "RemoveContainer" containerID="941d9c46e85baf3367574fd717e4e3dcdc8ad25d54f9e8dd9b67c65475cbc567"
Jul 16 20:01:47.771827 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.771806    2576 scope.go:117] "RemoveContainer" containerID="941d9c46e85baf3367574fd717e4e3dcdc8ad25d54f9e8dd9b67c65475cbc567"
Jul 16 20:01:47.772188 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:01:47.772155    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941d9c46e85baf3367574fd717e4e3dcdc8ad25d54f9e8dd9b67c65475cbc567\": container with ID starting with 941d9c46e85baf3367574fd717e4e3dcdc8ad25d54f9e8dd9b67c65475cbc567 not found: ID does not exist" containerID="941d9c46e85baf3367574fd717e4e3dcdc8ad25d54f9e8dd9b67c65475cbc567"
Jul 16 20:01:47.772291 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.772198    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941d9c46e85baf3367574fd717e4e3dcdc8ad25d54f9e8dd9b67c65475cbc567"} err="failed to get container status \"941d9c46e85baf3367574fd717e4e3dcdc8ad25d54f9e8dd9b67c65475cbc567\": rpc error: code = NotFound desc = could not find container \"941d9c46e85baf3367574fd717e4e3dcdc8ad25d54f9e8dd9b67c65475cbc567\": container with ID starting with 941d9c46e85baf3367574fd717e4e3dcdc8ad25d54f9e8dd9b67c65475cbc567 not found: ID does not exist"
Jul 16 20:01:47.821186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.821149    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-ca-trust-extracted\") pod \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") "
Jul 16 20:01:47.821370 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.821202    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-certificates\") pod \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") "
Jul 16 20:01:47.821370 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.821241    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nh5x\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-kube-api-access-7nh5x\") pod \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") "
Jul 16 20:01:47.821370 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.821265    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-trusted-ca\") pod \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") "
Jul 16 20:01:47.821370 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.821292    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-image-registry-private-configuration\") pod \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") "
Jul 16 20:01:47.821370 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.821334    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-bound-sa-token\") pod \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") "
Jul 16 20:01:47.821370 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.821365    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-installation-pull-secrets\") pod \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") "
Jul 16 20:01:47.821633 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.821393    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls\") pod \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\" (UID: \"aa58ca0b-1f88-4f88-a06c-a93efba1fd37\") "
Jul 16 20:01:47.826483 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.822462    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "aa58ca0b-1f88-4f88-a06c-a93efba1fd37" (UID: "aa58ca0b-1f88-4f88-a06c-a93efba1fd37"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:01:47.826483 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.825215    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "aa58ca0b-1f88-4f88-a06c-a93efba1fd37" (UID: "aa58ca0b-1f88-4f88-a06c-a93efba1fd37"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:01:47.826483 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.826207    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-kube-api-access-7nh5x" (OuterVolumeSpecName: "kube-api-access-7nh5x") pod "aa58ca0b-1f88-4f88-a06c-a93efba1fd37" (UID: "aa58ca0b-1f88-4f88-a06c-a93efba1fd37"). InnerVolumeSpecName "kube-api-access-7nh5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:01:47.827931 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.827895    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "aa58ca0b-1f88-4f88-a06c-a93efba1fd37" (UID: "aa58ca0b-1f88-4f88-a06c-a93efba1fd37"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:01:47.830981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.830111    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "aa58ca0b-1f88-4f88-a06c-a93efba1fd37" (UID: "aa58ca0b-1f88-4f88-a06c-a93efba1fd37"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:01:47.831830 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.831799    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "aa58ca0b-1f88-4f88-a06c-a93efba1fd37" (UID: "aa58ca0b-1f88-4f88-a06c-a93efba1fd37"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:01:47.840287 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.839903    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "aa58ca0b-1f88-4f88-a06c-a93efba1fd37" (UID: "aa58ca0b-1f88-4f88-a06c-a93efba1fd37"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:01:47.840287 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.840146    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "aa58ca0b-1f88-4f88-a06c-a93efba1fd37" (UID: "aa58ca0b-1f88-4f88-a06c-a93efba1fd37"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:01:47.922848 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.922780    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-ca-trust-extracted\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:01:47.922848 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.922806    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-certificates\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:01:47.922848 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.922815    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nh5x\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-kube-api-access-7nh5x\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:01:47.922848 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.922827    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-trusted-ca\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:01:47.922848 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.922844    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-image-registry-private-configuration\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:01:47.923139 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.922857    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-bound-sa-token\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:01:47.923139 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.922872    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-installation-pull-secrets\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:01:47.923139 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:47.922886    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa58ca0b-1f88-4f88-a06c-a93efba1fd37-registry-tls\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:01:48.107168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.107096    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-ff7887449-bq987"]
Jul 16 20:01:48.136732 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.131722    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-ff7887449-bq987"]
Jul 16 20:01:48.150928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.150889    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m62d4"]
Jul 16 20:01:48.151289 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.151272    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa58ca0b-1f88-4f88-a06c-a93efba1fd37" containerName="registry"
Jul 16 20:01:48.151361 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.151291    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa58ca0b-1f88-4f88-a06c-a93efba1fd37" containerName="registry"
Jul 16 20:01:48.151361 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.151350    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa58ca0b-1f88-4f88-a06c-a93efba1fd37" containerName="registry"
Jul 16 20:01:48.176319 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.176241    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.179305 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.179275    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 20:01:48.179449 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.179315    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 20:01:48.179449 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.179423    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-66zdp\""
Jul 16 20:01:48.179449 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.179325    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 20:01:48.244995 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.244961    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa58ca0b-1f88-4f88-a06c-a93efba1fd37" path="/var/lib/kubelet/pods/aa58ca0b-1f88-4f88-a06c-a93efba1fd37/volumes"
Jul 16 20:01:48.326671 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.326629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-textfile\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.326671 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.326679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfszk\" (UniqueName: \"kubernetes.io/projected/f58e77e5-5e72-4ef6-a574-abfa4326dae4-kube-api-access-kfszk\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.326933 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.326755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-wtmp\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.326933 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.326785    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f58e77e5-5e72-4ef6-a574-abfa4326dae4-metrics-client-ca\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.326933 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.326810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-tls\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.326933 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.326840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.326933 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.326875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f58e77e5-5e72-4ef6-a574-abfa4326dae4-root\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.326933 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.326906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-accelerators-collector-config\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.327218 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.326945    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f58e77e5-5e72-4ef6-a574-abfa4326dae4-sys\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.427916 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.427829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-wtmp\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.427916 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.427886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f58e77e5-5e72-4ef6-a574-abfa4326dae4-metrics-client-ca\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.427916 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.427910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-tls\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.428476 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.427939    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.428476 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.427972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f58e77e5-5e72-4ef6-a574-abfa4326dae4-root\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.428476 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.428002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-accelerators-collector-config\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.428476 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.428040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f58e77e5-5e72-4ef6-a574-abfa4326dae4-sys\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.428476 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.428066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-textfile\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.428476 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.428089    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfszk\" (UniqueName: \"kubernetes.io/projected/f58e77e5-5e72-4ef6-a574-abfa4326dae4-kube-api-access-kfszk\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.428800 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.428556    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-wtmp\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.429173 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.429146    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f58e77e5-5e72-4ef6-a574-abfa4326dae4-metrics-client-ca\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.429789 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.429617    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f58e77e5-5e72-4ef6-a574-abfa4326dae4-root\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.429789 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.429648    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f58e77e5-5e72-4ef6-a574-abfa4326dae4-sys\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.429995 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.429902    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-textfile\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.429995 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.429950    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-accelerators-collector-config\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.432459 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.432437    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-tls\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.432674 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.432649    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f58e77e5-5e72-4ef6-a574-abfa4326dae4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.442919 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.442888    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfszk\" (UniqueName: \"kubernetes.io/projected/f58e77e5-5e72-4ef6-a574-abfa4326dae4-kube-api-access-kfszk\") pod \"node-exporter-m62d4\" (UID: \"f58e77e5-5e72-4ef6-a574-abfa4326dae4\") " pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.488929 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.488892    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m62d4"
Jul 16 20:01:48.524727 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:01:48.521878    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58e77e5_5e72_4ef6_a574_abfa4326dae4.slice/crio-c6e8f756b92245f8c8fc22abb2bc014671752c9e5ddcbb656078bf78b904d98a WatchSource:0}: Error finding container c6e8f756b92245f8c8fc22abb2bc014671752c9e5ddcbb656078bf78b904d98a: Status 404 returned error can't find the container with id c6e8f756b92245f8c8fc22abb2bc014671752c9e5ddcbb656078bf78b904d98a
Jul 16 20:01:48.767987 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:48.767905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m62d4" event={"ID":"f58e77e5-5e72-4ef6-a574-abfa4326dae4","Type":"ContainerStarted","Data":"c6e8f756b92245f8c8fc22abb2bc014671752c9e5ddcbb656078bf78b904d98a"}
Jul 16 20:01:51.783575 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:51.783538    2576 generic.go:356] "Generic (PLEG): container finished" podID="f58e77e5-5e72-4ef6-a574-abfa4326dae4" containerID="372da591bda72c327b13f6a9b9ee9e9e32bafbd682f17130f40b15b990da0049" exitCode=0
Jul 16 20:01:51.784116 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:51.783606    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m62d4" event={"ID":"f58e77e5-5e72-4ef6-a574-abfa4326dae4","Type":"ContainerDied","Data":"372da591bda72c327b13f6a9b9ee9e9e32bafbd682f17130f40b15b990da0049"}
Jul 16 20:01:52.790633 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:52.790589    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m62d4" event={"ID":"f58e77e5-5e72-4ef6-a574-abfa4326dae4","Type":"ContainerStarted","Data":"c8799464a32e63f8db2a0b782a9d6e7071f044f04cf64658afa8d3d84a64f2d4"}
Jul 16 20:01:52.791198 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:52.790662    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m62d4" event={"ID":"f58e77e5-5e72-4ef6-a574-abfa4326dae4","Type":"ContainerStarted","Data":"6289032f80d2cd9f96a9ee2bcab661c99cdc1591380389cf4af743b4416c2739"}
Jul 16 20:01:52.811778 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:52.811727    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m62d4" podStartSLOduration=2.287105693 podStartE2EDuration="4.811690961s" totalImagesPullingTime="2.412717596s" totalInitContainerRuntime="111.867672ms" isStatefulPod=true podCreationTimestamp="2026-07-16 20:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:52.80976188 +0000 UTC m=+91.060382640" watchObservedRunningTime="2026-07-16 20:01:52.811690961 +0000 UTC m=+91.062311721"
Jul 16 20:01:54.341484 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.341444    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 20:01:54.371091 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.371053    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 20:01:54.371268 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.371166    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.374590 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.374369    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 20:01:54.374590 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.374407    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 20:01:54.374832 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.374596    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 20:01:54.374832 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.374621    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 20:01:54.375518 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.375118    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 20:01:54.375518 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.375142    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 20:01:54.375518 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.375150    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 20:01:54.375518 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.375162    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4cluc5u5ls9nl\""
Jul 16 20:01:54.375518 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.375119    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 20:01:54.375518 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.375413    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 20:01:54.375879 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.375560    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 20:01:54.375879 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.375586    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 20:01:54.375879 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.375560    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4497n\""
Jul 16 20:01:54.376020 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.376005    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 20:01:54.376234 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.376218    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 20:01:54.376655 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.376630    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.376745 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.376671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.376791 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.376765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.376838 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.376805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.376838 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.376829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.376942 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.376858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-config\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.376942 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.376908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.377048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.376940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.377048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.376984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.377048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.377011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49c8j\" (UniqueName: \"kubernetes.io/projected/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-kube-api-access-49c8j\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.377048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.377037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.377235 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.377109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.377235 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.377144    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.377235 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.377190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.377235 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.377213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.377423 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.377245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.377423 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.377322    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.377423 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.377351    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.377423 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.377383    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.377423 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.377413    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.378375 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.378353    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 20:01:54.381988 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.381922    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 20:01:54.478723 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.478669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.478899 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.478736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.478899 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.478760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.478899 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.478779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.478899 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.478811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.478899 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.478833    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.478899 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.478850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.478899 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.478873    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.478899 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.478898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.479267 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.478924    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.479267 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.478951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.479267 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.478978    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.479267 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.479015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.479267 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.479034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.479267 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.479059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-config\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.479267 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.479083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.479267 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.479105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.479267 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.479135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.479267 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.479144    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.479267 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.479161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49c8j\" (UniqueName: \"kubernetes.io/projected/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-kube-api-access-49c8j\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.479267 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.479221    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.480047 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.479550    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.480047 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.479861    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.481480 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.481451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.483726 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.482309    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.483726 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.482718    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.483726 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.482778    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.483726 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.482925    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.483726 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.483144    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.483726 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.483235    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.483726 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.483624    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.484122 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.483796    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.484193 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.484168    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.485791 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.485568    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.485791 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.485658    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.485791 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.485737    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.486017 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.485895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.486611 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.486568    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.487105 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.487085    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-config\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.488412 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.488371    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49c8j\" (UniqueName: \"kubernetes.io/projected/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-kube-api-access-49c8j\") pod \"prometheus-k8s-0\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.686259 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.686166    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:01:54.856588 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:54.856547    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 20:01:54.858719 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:01:54.858663    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa4539a_ad13_4607_a5cd_6b799ca5d0f3.slice/crio-3b6dc0419ab29ba56d7c848dabd0bad6db704b9fe1645a1177a947acebba45c2 WatchSource:0}: Error finding container 3b6dc0419ab29ba56d7c848dabd0bad6db704b9fe1645a1177a947acebba45c2: Status 404 returned error can't find the container with id 3b6dc0419ab29ba56d7c848dabd0bad6db704b9fe1645a1177a947acebba45c2
Jul 16 20:01:55.804841 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:55.804795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerStarted","Data":"3b6dc0419ab29ba56d7c848dabd0bad6db704b9fe1645a1177a947acebba45c2"}
Jul 16 20:01:56.809195 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:56.809093    2576 generic.go:356] "Generic (PLEG): container finished" podID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerID="2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822" exitCode=0
Jul 16 20:01:56.809195 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:56.809182    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerDied","Data":"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"}
Jul 16 20:01:57.660305 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.660268    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69cf4fb89f-fxb2x"]
Jul 16 20:01:57.676854 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.676825    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.677086 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.677056    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69cf4fb89f-fxb2x"]
Jul 16 20:01:57.686478 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.686294    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 16 20:01:57.710037 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.709963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67b1a72c-cc4b-4554-8871-c0a857781b75-console-oauth-config\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.710037 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.710009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm7bq\" (UniqueName: \"kubernetes.io/projected/67b1a72c-cc4b-4554-8871-c0a857781b75-kube-api-access-wm7bq\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.710264 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.710043    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67b1a72c-cc4b-4554-8871-c0a857781b75-console-serving-cert\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.710264 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.710076    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-oauth-serving-cert\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.710264 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.710200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-trusted-ca-bundle\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.710264 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.710242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-console-config\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.710408 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.710268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-service-ca\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.810891 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.810857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67b1a72c-cc4b-4554-8871-c0a857781b75-console-oauth-config\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.811337 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.810912    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm7bq\" (UniqueName: \"kubernetes.io/projected/67b1a72c-cc4b-4554-8871-c0a857781b75-kube-api-access-wm7bq\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.811337 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.810942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67b1a72c-cc4b-4554-8871-c0a857781b75-console-serving-cert\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.811337 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.810977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-oauth-serving-cert\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.811337 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.811028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-trusted-ca-bundle\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.811337 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.811054    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-console-config\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.811337 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.811076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-service-ca\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.814069 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.811930    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-oauth-serving-cert\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.814069 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.811970    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-console-config\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.814069 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.812372    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-trusted-ca-bundle\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.814069 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.812858    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-service-ca\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.814352 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.814316    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67b1a72c-cc4b-4554-8871-c0a857781b75-console-oauth-config\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.819578 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.819552    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67b1a72c-cc4b-4554-8871-c0a857781b75-console-serving-cert\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.821744 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.821675    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm7bq\" (UniqueName: \"kubernetes.io/projected/67b1a72c-cc4b-4554-8871-c0a857781b75-kube-api-access-wm7bq\") pod \"console-69cf4fb89f-fxb2x\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") " pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:57.989840 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:57.989756    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:01:58.163365 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:58.163335    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69cf4fb89f-fxb2x"]
Jul 16 20:01:58.166470 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:01:58.166440    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b1a72c_cc4b_4554_8871_c0a857781b75.slice/crio-2a8449e26318722ba80e513cbfc79a74ecfca1011fe2da1600d5f0621bc6605f WatchSource:0}: Error finding container 2a8449e26318722ba80e513cbfc79a74ecfca1011fe2da1600d5f0621bc6605f: Status 404 returned error can't find the container with id 2a8449e26318722ba80e513cbfc79a74ecfca1011fe2da1600d5f0621bc6605f
Jul 16 20:01:58.824900 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:58.824862    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69cf4fb89f-fxb2x" event={"ID":"67b1a72c-cc4b-4554-8871-c0a857781b75","Type":"ContainerStarted","Data":"0977467fd9a1ca6f3a897119574638b040b7fb0f6932a45c9eed6e2a8791bdb3"}
Jul 16 20:01:58.825306 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:58.824921    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69cf4fb89f-fxb2x" event={"ID":"67b1a72c-cc4b-4554-8871-c0a857781b75","Type":"ContainerStarted","Data":"2a8449e26318722ba80e513cbfc79a74ecfca1011fe2da1600d5f0621bc6605f"}
Jul 16 20:01:58.845008 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:01:58.844962    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69cf4fb89f-fxb2x" podStartSLOduration=1.844945914 podStartE2EDuration="1.844945914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:01:58.843599505 +0000 UTC m=+97.094220263" watchObservedRunningTime="2026-07-16 20:01:58.844945914 +0000 UTC m=+97.095566675"
Jul 16 20:02:01.840256 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:01.840216    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerStarted","Data":"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"}
Jul 16 20:02:01.840658 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:01.840325    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerStarted","Data":"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"}
Jul 16 20:02:03.851781 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:03.851745    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerStarted","Data":"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"}
Jul 16 20:02:03.852162 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:03.851838    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerStarted","Data":"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"}
Jul 16 20:02:03.852162 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:03.851870    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerStarted","Data":"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"}
Jul 16 20:02:03.852162 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:03.851896    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerStarted","Data":"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"}
Jul 16 20:02:03.880981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:03.880916    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.041096838 podStartE2EDuration="9.880895318s" totalImagesPullingTime="7.817381019s" totalInitContainerRuntime="22.417461ms" isStatefulPod=false podCreationTimestamp="2026-07-16 20:01:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:02:03.879253926 +0000 UTC m=+102.129874685" watchObservedRunningTime="2026-07-16 20:02:03.880895318 +0000 UTC m=+102.131516077"
Jul 16 20:02:04.686438 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:04.686404    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:02:07.795557 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:07.795516    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d7fd8d5df-4xzd8" podUID="904f4feb-d288-4671-a63b-9109773aaf56" containerName="console" containerID="cri-o://8a33c46886ac63e56902acc28b44f5afdbdd4cd36308387e23bb034e398b3ff9" gracePeriod=15
Jul 16 20:02:07.990303 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:07.990262    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:02:07.990446 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:07.990346    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:02:07.995149 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:07.995126    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:02:08.071986 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.071962    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:02:08.209439 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.209409    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-console-config\") pod \"904f4feb-d288-4671-a63b-9109773aaf56\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") "
Jul 16 20:02:08.209615 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.209468    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hldv\" (UniqueName: \"kubernetes.io/projected/904f4feb-d288-4671-a63b-9109773aaf56-kube-api-access-2hldv\") pod \"904f4feb-d288-4671-a63b-9109773aaf56\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") "
Jul 16 20:02:08.209615 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.209494    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/904f4feb-d288-4671-a63b-9109773aaf56-console-serving-cert\") pod \"904f4feb-d288-4671-a63b-9109773aaf56\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") "
Jul 16 20:02:08.209615 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.209513    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-oauth-serving-cert\") pod \"904f4feb-d288-4671-a63b-9109773aaf56\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") "
Jul 16 20:02:08.209615 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.209542    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/904f4feb-d288-4671-a63b-9109773aaf56-console-oauth-config\") pod \"904f4feb-d288-4671-a63b-9109773aaf56\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") "
Jul 16 20:02:08.209615 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.209570    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-service-ca\") pod \"904f4feb-d288-4671-a63b-9109773aaf56\" (UID: \"904f4feb-d288-4671-a63b-9109773aaf56\") "
Jul 16 20:02:08.209956 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.209837    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-console-config" (OuterVolumeSpecName: "console-config") pod "904f4feb-d288-4671-a63b-9109773aaf56" (UID: "904f4feb-d288-4671-a63b-9109773aaf56"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:02:08.209956 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.209943    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "904f4feb-d288-4671-a63b-9109773aaf56" (UID: "904f4feb-d288-4671-a63b-9109773aaf56"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:02:08.210061 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.210044    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-service-ca" (OuterVolumeSpecName: "service-ca") pod "904f4feb-d288-4671-a63b-9109773aaf56" (UID: "904f4feb-d288-4671-a63b-9109773aaf56"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:02:08.212107 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.212081    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904f4feb-d288-4671-a63b-9109773aaf56-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "904f4feb-d288-4671-a63b-9109773aaf56" (UID: "904f4feb-d288-4671-a63b-9109773aaf56"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:02:08.212328 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.212308    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904f4feb-d288-4671-a63b-9109773aaf56-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "904f4feb-d288-4671-a63b-9109773aaf56" (UID: "904f4feb-d288-4671-a63b-9109773aaf56"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:02:08.212328 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.212309    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904f4feb-d288-4671-a63b-9109773aaf56-kube-api-access-2hldv" (OuterVolumeSpecName: "kube-api-access-2hldv") pod "904f4feb-d288-4671-a63b-9109773aaf56" (UID: "904f4feb-d288-4671-a63b-9109773aaf56"). InnerVolumeSpecName "kube-api-access-2hldv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:02:08.310405 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.310377    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/904f4feb-d288-4671-a63b-9109773aaf56-console-oauth-config\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:02:08.310405 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.310401    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-service-ca\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:02:08.310631 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.310417    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-console-config\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:02:08.310631 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.310429    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hldv\" (UniqueName: \"kubernetes.io/projected/904f4feb-d288-4671-a63b-9109773aaf56-kube-api-access-2hldv\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:02:08.310631 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.310438    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/904f4feb-d288-4671-a63b-9109773aaf56-console-serving-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:02:08.310631 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.310446    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/904f4feb-d288-4671-a63b-9109773aaf56-oauth-serving-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:02:08.870163 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.870123    2576 generic.go:356] "Generic (PLEG): container finished" podID="904f4feb-d288-4671-a63b-9109773aaf56" containerID="8a33c46886ac63e56902acc28b44f5afdbdd4cd36308387e23bb034e398b3ff9" exitCode=0
Jul 16 20:02:08.870669 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.870175    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7fd8d5df-4xzd8"
Jul 16 20:02:08.870669 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.870207    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7fd8d5df-4xzd8" event={"ID":"904f4feb-d288-4671-a63b-9109773aaf56","Type":"ContainerDied","Data":"8a33c46886ac63e56902acc28b44f5afdbdd4cd36308387e23bb034e398b3ff9"}
Jul 16 20:02:08.870669 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.870259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7fd8d5df-4xzd8" event={"ID":"904f4feb-d288-4671-a63b-9109773aaf56","Type":"ContainerDied","Data":"d6ab2bd39367a44b7cd8a0bc9bc3222d9feb7f2633bceb9aea86863cdbb5f03d"}
Jul 16 20:02:08.870669 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.870281    2576 scope.go:117] "RemoveContainer" containerID="8a33c46886ac63e56902acc28b44f5afdbdd4cd36308387e23bb034e398b3ff9"
Jul 16 20:02:08.874764 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.874746    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:02:08.878458 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.878444    2576 scope.go:117] "RemoveContainer" containerID="8a33c46886ac63e56902acc28b44f5afdbdd4cd36308387e23bb034e398b3ff9"
Jul 16 20:02:08.878774 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:02:08.878751    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a33c46886ac63e56902acc28b44f5afdbdd4cd36308387e23bb034e398b3ff9\": container with ID starting with 8a33c46886ac63e56902acc28b44f5afdbdd4cd36308387e23bb034e398b3ff9 not found: ID does not exist" containerID="8a33c46886ac63e56902acc28b44f5afdbdd4cd36308387e23bb034e398b3ff9"
Jul 16 20:02:08.878826 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.878788    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a33c46886ac63e56902acc28b44f5afdbdd4cd36308387e23bb034e398b3ff9"} err="failed to get container status \"8a33c46886ac63e56902acc28b44f5afdbdd4cd36308387e23bb034e398b3ff9\": rpc error: code = NotFound desc = could not find container \"8a33c46886ac63e56902acc28b44f5afdbdd4cd36308387e23bb034e398b3ff9\": container with ID starting with 8a33c46886ac63e56902acc28b44f5afdbdd4cd36308387e23bb034e398b3ff9 not found: ID does not exist"
Jul 16 20:02:08.885564 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.885545    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d7fd8d5df-4xzd8"]
Jul 16 20:02:08.889081 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:08.889062    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d7fd8d5df-4xzd8"]
Jul 16 20:02:10.243389 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:10.243360    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="904f4feb-d288-4671-a63b-9109773aaf56" path="/var/lib/kubelet/pods/904f4feb-d288-4671-a63b-9109773aaf56/volumes"
Jul 16 20:02:15.755535 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:15.755498    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7jgvl"
Jul 16 20:02:21.324182 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:21.324138    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l4947_bb00e945-1b1b-4387-8834-cd0365bdcd23/serve-healthcheck-canary/0.log"
Jul 16 20:02:54.687219 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:54.687188    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:02:54.706661 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:54.706637    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:02:55.049198 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:02:55.049115    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:12.684565 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:12.684478    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 20:03:12.685140 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:12.684890    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="prometheus" containerID="cri-o://adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417" gracePeriod=600
Jul 16 20:03:12.685249 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:12.685221    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8" gracePeriod=600
Jul 16 20:03:12.686251 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:12.685771    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="config-reloader" containerID="cri-o://da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768" gracePeriod=600
Jul 16 20:03:12.686251 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:12.685639    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="thanos-sidecar" containerID="cri-o://0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd" gracePeriod=600
Jul 16 20:03:12.686251 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:12.685669    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="kube-rbac-proxy" containerID="cri-o://9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb" gracePeriod=600
Jul 16 20:03:12.686251 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:12.685639    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="kube-rbac-proxy-web" containerID="cri-o://a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f" gracePeriod=600
Jul 16 20:03:12.940380 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:12.940312    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.025656 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.025627    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-web-config\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.025850 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.025675    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-tls\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.025850 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.025720    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-metrics-client-certs\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.025850 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.025750    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49c8j\" (UniqueName: \"kubernetes.io/projected/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-kube-api-access-49c8j\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.025850 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.025768    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-tls-assets\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.025850 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.025795    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-metrics-client-ca\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.025850 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.025817    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-config\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.025850 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.025839    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-1\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.026219 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.025873    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.026219 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.025899    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-0\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.026219 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.025960    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-thanos-prometheus-http-client-file\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.026219 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.025986    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-db\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.026219 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.026018    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-trusted-ca-bundle\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.026219 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.026052    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-serving-certs-ca-bundle\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.026219 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.026095    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-kubelet-serving-ca-bundle\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.026219 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.026125    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-kube-rbac-proxy\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.026219 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.026160    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.026833 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.026783    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:03:13.028581 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.027680    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:03:13.028581 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.028016    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:03:13.029474 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.028791    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-grpc-tls\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.029474 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.028849    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-2\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.029474 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.028875    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-config-out\") pod \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\" (UID: \"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3\") "
Jul 16 20:03:13.029474 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.028988    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:03:13.029474 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.029251    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:03:13.029474 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.029267    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:03:13.029879 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.029508    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:03:13.029879 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.029532    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-db\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.029879 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.029552    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.029879 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.029567    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.029879 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.029582    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.029879 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.029600    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-configmap-metrics-client-ca\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.029879 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.029618    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.029879 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.029746    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:03:13.030817 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.030356    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:03:13.030817 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.030465    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-kube-api-access-49c8j" (OuterVolumeSpecName: "kube-api-access-49c8j") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "kube-api-access-49c8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:03:13.030817 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.030541    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:03:13.030817 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.030591    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-config" (OuterVolumeSpecName: "config") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:03:13.031078 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.030955    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:03:13.031078 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.031038    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:03:13.031480 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.031453    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:03:13.031618 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.031599    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-config-out" (OuterVolumeSpecName: "config-out") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:03:13.032035 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.032005    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:03:13.032160 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.032142    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:03:13.032209 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.032165    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:03:13.039870 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.039852    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-web-config" (OuterVolumeSpecName: "web-config") pod "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" (UID: "0fa4539a-ad13-4607-a5cd-6b799ca5d0f3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:03:13.097327 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097295    2576 generic.go:356] "Generic (PLEG): container finished" podID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerID="9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8" exitCode=0
Jul 16 20:03:13.097327 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097325    2576 generic.go:356] "Generic (PLEG): container finished" podID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerID="9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb" exitCode=0
Jul 16 20:03:13.097485 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097334    2576 generic.go:356] "Generic (PLEG): container finished" podID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerID="a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f" exitCode=0
Jul 16 20:03:13.097485 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097343    2576 generic.go:356] "Generic (PLEG): container finished" podID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerID="0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd" exitCode=0
Jul 16 20:03:13.097485 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097352    2576 generic.go:356] "Generic (PLEG): container finished" podID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerID="da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768" exitCode=0
Jul 16 20:03:13.097485 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097360    2576 generic.go:356] "Generic (PLEG): container finished" podID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerID="adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417" exitCode=0
Jul 16 20:03:13.097485 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097368    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerDied","Data":"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"}
Jul 16 20:03:13.097485 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097401    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.097485 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097435    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerDied","Data":"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"}
Jul 16 20:03:13.097485 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097463    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerDied","Data":"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"}
Jul 16 20:03:13.097485 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097489    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerDied","Data":"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"}
Jul 16 20:03:13.097783 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097508    2576 scope.go:117] "RemoveContainer" containerID="9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"
Jul 16 20:03:13.097783 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097514    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerDied","Data":"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"}
Jul 16 20:03:13.097783 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097538    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerDied","Data":"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"}
Jul 16 20:03:13.097783 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.097561    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fa4539a-ad13-4607-a5cd-6b799ca5d0f3","Type":"ContainerDied","Data":"3b6dc0419ab29ba56d7c848dabd0bad6db704b9fe1645a1177a947acebba45c2"}
Jul 16 20:03:13.106354 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.105908    2576 scope.go:117] "RemoveContainer" containerID="9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"
Jul 16 20:03:13.113277 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.113262    2576 scope.go:117] "RemoveContainer" containerID="a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"
Jul 16 20:03:13.120346 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.120325    2576 scope.go:117] "RemoveContainer" containerID="0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"
Jul 16 20:03:13.121512 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.121493    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 20:03:13.127082 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.127067    2576 scope.go:117] "RemoveContainer" containerID="da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"
Jul 16 20:03:13.128252 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.128234    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 20:03:13.130268 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130246    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.130346 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130267    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.130346 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130284    2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.130346 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130298    2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-kube-rbac-proxy\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.130346 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130312    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.130346 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130327    2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-grpc-tls\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.130346 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130338    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.130346 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130347    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-config-out\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.130591 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130356    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-web-config\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.130591 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130364    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.130591 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130371    2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-secret-metrics-client-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.130591 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130380    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49c8j\" (UniqueName: \"kubernetes.io/projected/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-kube-api-access-49c8j\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.130591 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130387    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-tls-assets\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.130591 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.130395    2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3-config\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:13.133808 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.133780    2576 scope.go:117] "RemoveContainer" containerID="adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"
Jul 16 20:03:13.140551 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.140536    2576 scope.go:117] "RemoveContainer" containerID="2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"
Jul 16 20:03:13.146874 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.146851    2576 scope.go:117] "RemoveContainer" containerID="9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"
Jul 16 20:03:13.147137 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:03:13.147116    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8\": container with ID starting with 9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8 not found: ID does not exist" containerID="9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"
Jul 16 20:03:13.147213 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.147149    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"} err="failed to get container status \"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8\": rpc error: code = NotFound desc = could not find container \"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8\": container with ID starting with 9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8 not found: ID does not exist"
Jul 16 20:03:13.147213 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.147174    2576 scope.go:117] "RemoveContainer" containerID="9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"
Jul 16 20:03:13.147427 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:03:13.147411    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb\": container with ID starting with 9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb not found: ID does not exist" containerID="9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"
Jul 16 20:03:13.147470 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.147436    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"} err="failed to get container status \"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb\": rpc error: code = NotFound desc = could not find container \"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb\": container with ID starting with 9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb not found: ID does not exist"
Jul 16 20:03:13.147470 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.147455    2576 scope.go:117] "RemoveContainer" containerID="a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"
Jul 16 20:03:13.147722 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:03:13.147683    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f\": container with ID starting with a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f not found: ID does not exist" containerID="a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"
Jul 16 20:03:13.147769 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.147722    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"} err="failed to get container status \"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f\": rpc error: code = NotFound desc = could not find container \"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f\": container with ID starting with a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f not found: ID does not exist"
Jul 16 20:03:13.147769 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.147742    2576 scope.go:117] "RemoveContainer" containerID="0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"
Jul 16 20:03:13.147962 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:03:13.147946    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd\": container with ID starting with 0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd not found: ID does not exist" containerID="0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"
Jul 16 20:03:13.147999 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.147968    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"} err="failed to get container status \"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd\": rpc error: code = NotFound desc = could not find container \"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd\": container with ID starting with 0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd not found: ID does not exist"
Jul 16 20:03:13.147999 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.147983    2576 scope.go:117] "RemoveContainer" containerID="da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"
Jul 16 20:03:13.148211 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:03:13.148196    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768\": container with ID starting with da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768 not found: ID does not exist" containerID="da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"
Jul 16 20:03:13.148253 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.148215    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"} err="failed to get container status \"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768\": rpc error: code = NotFound desc = could not find container \"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768\": container with ID starting with da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768 not found: ID does not exist"
Jul 16 20:03:13.148253 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.148228    2576 scope.go:117] "RemoveContainer" containerID="adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"
Jul 16 20:03:13.148434 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:03:13.148421    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417\": container with ID starting with adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417 not found: ID does not exist" containerID="adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"
Jul 16 20:03:13.148478 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.148438    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"} err="failed to get container status \"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417\": rpc error: code = NotFound desc = could not find container \"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417\": container with ID starting with adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417 not found: ID does not exist"
Jul 16 20:03:13.148478 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.148451    2576 scope.go:117] "RemoveContainer" containerID="2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"
Jul 16 20:03:13.148639 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:03:13.148621    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822\": container with ID starting with 2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822 not found: ID does not exist" containerID="2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"
Jul 16 20:03:13.148675 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.148644    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"} err="failed to get container status \"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822\": rpc error: code = NotFound desc = could not find container \"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822\": container with ID starting with 2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822 not found: ID does not exist"
Jul 16 20:03:13.148675 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.148658    2576 scope.go:117] "RemoveContainer" containerID="9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"
Jul 16 20:03:13.148900 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.148883    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"} err="failed to get container status \"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8\": rpc error: code = NotFound desc = could not find container \"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8\": container with ID starting with 9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8 not found: ID does not exist"
Jul 16 20:03:13.148961 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.148901    2576 scope.go:117] "RemoveContainer" containerID="9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"
Jul 16 20:03:13.149099 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.149084    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"} err="failed to get container status \"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb\": rpc error: code = NotFound desc = could not find container \"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb\": container with ID starting with 9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb not found: ID does not exist"
Jul 16 20:03:13.149139 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.149100    2576 scope.go:117] "RemoveContainer" containerID="a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"
Jul 16 20:03:13.149319 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.149302    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"} err="failed to get container status \"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f\": rpc error: code = NotFound desc = could not find container \"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f\": container with ID starting with a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f not found: ID does not exist"
Jul 16 20:03:13.149371 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.149320    2576 scope.go:117] "RemoveContainer" containerID="0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"
Jul 16 20:03:13.149511 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.149495    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"} err="failed to get container status \"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd\": rpc error: code = NotFound desc = could not find container \"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd\": container with ID starting with 0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd not found: ID does not exist"
Jul 16 20:03:13.149548 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.149513    2576 scope.go:117] "RemoveContainer" containerID="da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"
Jul 16 20:03:13.149735 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.149719    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"} err="failed to get container status \"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768\": rpc error: code = NotFound desc = could not find container \"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768\": container with ID starting with da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768 not found: ID does not exist"
Jul 16 20:03:13.149785 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.149735    2576 scope.go:117] "RemoveContainer" containerID="adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"
Jul 16 20:03:13.149984 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.149968    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"} err="failed to get container status \"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417\": rpc error: code = NotFound desc = could not find container \"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417\": container with ID starting with adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417 not found: ID does not exist"
Jul 16 20:03:13.149984 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.149983    2576 scope.go:117] "RemoveContainer" containerID="2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"
Jul 16 20:03:13.150165 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.150149    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"} err="failed to get container status \"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822\": rpc error: code = NotFound desc = could not find container \"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822\": container with ID starting with 2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822 not found: ID does not exist"
Jul 16 20:03:13.150209 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.150166    2576 scope.go:117] "RemoveContainer" containerID="9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"
Jul 16 20:03:13.150382 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.150362    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"} err="failed to get container status \"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8\": rpc error: code = NotFound desc = could not find container \"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8\": container with ID starting with 9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8 not found: ID does not exist"
Jul 16 20:03:13.150432 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.150382    2576 scope.go:117] "RemoveContainer" containerID="9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"
Jul 16 20:03:13.150576 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.150561    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"} err="failed to get container status \"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb\": rpc error: code = NotFound desc = could not find container \"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb\": container with ID starting with 9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb not found: ID does not exist"
Jul 16 20:03:13.150617 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.150577    2576 scope.go:117] "RemoveContainer" containerID="a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"
Jul 16 20:03:13.150776 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.150758    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"} err="failed to get container status \"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f\": rpc error: code = NotFound desc = could not find container \"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f\": container with ID starting with a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f not found: ID does not exist"
Jul 16 20:03:13.150826 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.150778    2576 scope.go:117] "RemoveContainer" containerID="0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"
Jul 16 20:03:13.150980 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.150962    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"} err="failed to get container status \"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd\": rpc error: code = NotFound desc = could not find container \"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd\": container with ID starting with 0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd not found: ID does not exist"
Jul 16 20:03:13.151030 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.150981    2576 scope.go:117] "RemoveContainer" containerID="da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"
Jul 16 20:03:13.151196 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.151180    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"} err="failed to get container status \"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768\": rpc error: code = NotFound desc = could not find container \"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768\": container with ID starting with da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768 not found: ID does not exist"
Jul 16 20:03:13.151236 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.151197    2576 scope.go:117] "RemoveContainer" containerID="adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"
Jul 16 20:03:13.151386 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.151371    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"} err="failed to get container status \"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417\": rpc error: code = NotFound desc = could not find container \"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417\": container with ID starting with adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417 not found: ID does not exist"
Jul 16 20:03:13.151428 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.151386    2576 scope.go:117] "RemoveContainer" containerID="2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"
Jul 16 20:03:13.151577 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.151560    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"} err="failed to get container status \"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822\": rpc error: code = NotFound desc = could not find container \"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822\": container with ID starting with 2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822 not found: ID does not exist"
Jul 16 20:03:13.151616 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.151577    2576 scope.go:117] "RemoveContainer" containerID="9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"
Jul 16 20:03:13.151797 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.151781    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"} err="failed to get container status \"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8\": rpc error: code = NotFound desc = could not find container \"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8\": container with ID starting with 9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8 not found: ID does not exist"
Jul 16 20:03:13.151834 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.151797    2576 scope.go:117] "RemoveContainer" containerID="9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"
Jul 16 20:03:13.152011 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.151996    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"} err="failed to get container status \"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb\": rpc error: code = NotFound desc = could not find container \"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb\": container with ID starting with 9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb not found: ID does not exist"
Jul 16 20:03:13.152063 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.152011    2576 scope.go:117] "RemoveContainer" containerID="a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"
Jul 16 20:03:13.152191 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.152176    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"} err="failed to get container status \"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f\": rpc error: code = NotFound desc = could not find container \"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f\": container with ID starting with a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f not found: ID does not exist"
Jul 16 20:03:13.152230 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.152190    2576 scope.go:117] "RemoveContainer" containerID="0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"
Jul 16 20:03:13.152405 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.152387    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"} err="failed to get container status \"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd\": rpc error: code = NotFound desc = could not find container \"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd\": container with ID starting with 0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd not found: ID does not exist"
Jul 16 20:03:13.152445 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.152407    2576 scope.go:117] "RemoveContainer" containerID="da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"
Jul 16 20:03:13.152585 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.152571    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"} err="failed to get container status \"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768\": rpc error: code = NotFound desc = could not find container \"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768\": container with ID starting with da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768 not found: ID does not exist"
Jul 16 20:03:13.152629 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.152585    2576 scope.go:117] "RemoveContainer" containerID="adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"
Jul 16 20:03:13.152852 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.152834    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"} err="failed to get container status \"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417\": rpc error: code = NotFound desc = could not find container \"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417\": container with ID starting with adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417 not found: ID does not exist"
Jul 16 20:03:13.152852 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.152851    2576 scope.go:117] "RemoveContainer" containerID="2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"
Jul 16 20:03:13.153026 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.153011    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"} err="failed to get container status \"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822\": rpc error: code = NotFound desc = could not find container \"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822\": container with ID starting with 2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822 not found: ID does not exist"
Jul 16 20:03:13.153067 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.153026    2576 scope.go:117] "RemoveContainer" containerID="9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"
Jul 16 20:03:13.153204 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.153186    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"} err="failed to get container status \"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8\": rpc error: code = NotFound desc = could not find container \"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8\": container with ID starting with 9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8 not found: ID does not exist"
Jul 16 20:03:13.153243 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.153207    2576 scope.go:117] "RemoveContainer" containerID="9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"
Jul 16 20:03:13.153378 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.153360    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"} err="failed to get container status \"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb\": rpc error: code = NotFound desc = could not find container \"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb\": container with ID starting with 9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb not found: ID does not exist"
Jul 16 20:03:13.153417 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.153378    2576 scope.go:117] "RemoveContainer" containerID="a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"
Jul 16 20:03:13.153573 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.153556    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"} err="failed to get container status \"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f\": rpc error: code = NotFound desc = could not find container \"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f\": container with ID starting with a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f not found: ID does not exist"
Jul 16 20:03:13.153629 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.153574    2576 scope.go:117] "RemoveContainer" containerID="0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"
Jul 16 20:03:13.153751 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.153730    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"} err="failed to get container status \"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd\": rpc error: code = NotFound desc = could not find container \"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd\": container with ID starting with 0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd not found: ID does not exist"
Jul 16 20:03:13.153751 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.153750    2576 scope.go:117] "RemoveContainer" containerID="da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"
Jul 16 20:03:13.153993 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.153979    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"} err="failed to get container status \"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768\": rpc error: code = NotFound desc = could not find container \"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768\": container with ID starting with da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768 not found: ID does not exist"
Jul 16 20:03:13.153993 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.153992    2576 scope.go:117] "RemoveContainer" containerID="adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"
Jul 16 20:03:13.154165 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.154146    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"} err="failed to get container status \"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417\": rpc error: code = NotFound desc = could not find container \"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417\": container with ID starting with adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417 not found: ID does not exist"
Jul 16 20:03:13.154217 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.154167    2576 scope.go:117] "RemoveContainer" containerID="2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"
Jul 16 20:03:13.154367 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.154353    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"} err="failed to get container status \"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822\": rpc error: code = NotFound desc = could not find container \"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822\": container with ID starting with 2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822 not found: ID does not exist"
Jul 16 20:03:13.154411 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.154367    2576 scope.go:117] "RemoveContainer" containerID="9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"
Jul 16 20:03:13.154572 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.154557    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8"} err="failed to get container status \"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8\": rpc error: code = NotFound desc = could not find container \"9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8\": container with ID starting with 9ce9e86eead3114b3e5057d0db290d7bf2faaf3c86f598745fc33c5583eaefb8 not found: ID does not exist"
Jul 16 20:03:13.154616 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.154573    2576 scope.go:117] "RemoveContainer" containerID="9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"
Jul 16 20:03:13.154749 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.154730    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb"} err="failed to get container status \"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb\": rpc error: code = NotFound desc = could not find container \"9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb\": container with ID starting with 9e841208a4746bfc109d53330fd3cd162d11ab08a3c15e41529cf9f2510300fb not found: ID does not exist"
Jul 16 20:03:13.154805 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.154751    2576 scope.go:117] "RemoveContainer" containerID="a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"
Jul 16 20:03:13.154962 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.154944    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f"} err="failed to get container status \"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f\": rpc error: code = NotFound desc = could not find container \"a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f\": container with ID starting with a17255d04b7304b96ab303afba76b5670067bb1d4e6ed225caf45efa7703fe4f not found: ID does not exist"
Jul 16 20:03:13.154962 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.154962    2576 scope.go:117] "RemoveContainer" containerID="0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"
Jul 16 20:03:13.155148 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.155130    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd"} err="failed to get container status \"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd\": rpc error: code = NotFound desc = could not find container \"0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd\": container with ID starting with 0d59245053eca8f2d493f06255646d66be19e08982daa24c800a212c44bf3afd not found: ID does not exist"
Jul 16 20:03:13.155188 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.155149    2576 scope.go:117] "RemoveContainer" containerID="da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"
Jul 16 20:03:13.160300 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.159541    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768"} err="failed to get container status \"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768\": rpc error: code = NotFound desc = could not find container \"da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768\": container with ID starting with da49a27628accb07af0dfb1795f77b7f287bc2694de57abc3b0afc4ce0d61768 not found: ID does not exist"
Jul 16 20:03:13.160300 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.159576    2576 scope.go:117] "RemoveContainer" containerID="adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"
Jul 16 20:03:13.160872 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.160841    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417"} err="failed to get container status \"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417\": rpc error: code = NotFound desc = could not find container \"adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417\": container with ID starting with adda6e2c1f8a4d40bdf22846900564aa66a7db297b95592aa5f876ff0ed99417 not found: ID does not exist"
Jul 16 20:03:13.160961 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.160874    2576 scope.go:117] "RemoveContainer" containerID="2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"
Jul 16 20:03:13.161142 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161124    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822"} err="failed to get container status \"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822\": rpc error: code = NotFound desc = could not find container \"2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822\": container with ID starting with 2606ab5e127eacafdf6034b7e52ce2714f255990d9ecbb2ee2cfc042180d8822 not found: ID does not exist"
Jul 16 20:03:13.161503 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161487    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 20:03:13.161760 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161745    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="init-config-reloader"
Jul 16 20:03:13.161806 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161761    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="init-config-reloader"
Jul 16 20:03:13.161806 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161768    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="prometheus"
Jul 16 20:03:13.161806 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161773    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="prometheus"
Jul 16 20:03:13.161806 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161780    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="config-reloader"
Jul 16 20:03:13.161806 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161785    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="config-reloader"
Jul 16 20:03:13.161806 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161791    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="kube-rbac-proxy-web"
Jul 16 20:03:13.161806 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161797    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="kube-rbac-proxy-web"
Jul 16 20:03:13.161806 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161806    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="thanos-sidecar"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161811    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="thanos-sidecar"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161818    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="kube-rbac-proxy"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161823    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="kube-rbac-proxy"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161828    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="kube-rbac-proxy-thanos"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161834    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="kube-rbac-proxy-thanos"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161842    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904f4feb-d288-4671-a63b-9109773aaf56" containerName="console"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161846    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="904f4feb-d288-4671-a63b-9109773aaf56" containerName="console"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161883    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="prometheus"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161888    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="kube-rbac-proxy-thanos"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161895    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="904f4feb-d288-4671-a63b-9109773aaf56" containerName="console"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161900    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="config-reloader"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161905    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="kube-rbac-proxy"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161910    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="thanos-sidecar"
Jul 16 20:03:13.162115 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.161916    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" containerName="kube-rbac-proxy-web"
Jul 16 20:03:13.165356 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.165341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.167741 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.167724    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 20:03:13.167838 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.167787    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 20:03:13.167897 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.167850    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 20:03:13.168105 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.168088    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 20:03:13.168170 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.168116    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 20:03:13.168170 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.168142    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 20:03:13.168170 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.168159    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 20:03:13.168314 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.168163    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4497n\""
Jul 16 20:03:13.168314 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.168308    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 20:03:13.168396 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.168322    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 20:03:13.168396 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.168332    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 20:03:13.169877 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.169847    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 20:03:13.169953 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.169877    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 20:03:13.169953 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.169884    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4cluc5u5ls9nl\""
Jul 16 20:03:13.170256 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.170243    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 20:03:13.172600 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.172534    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 20:03:13.174924 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.174907    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 20:03:13.181116 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.181097    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 20:03:13.231670 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.231670 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.231670 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-config\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.231872 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.231872 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5dw7\" (UniqueName: \"kubernetes.io/projected/bac34282-8050-4d86-8ab1-b2823b7fa1a5-kube-api-access-l5dw7\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.231872 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bac34282-8050-4d86-8ab1-b2823b7fa1a5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.231872 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.231872 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-web-config\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.231872 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.231872 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.232072 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.232072 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.232072 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231944    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.232072 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.231993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bac34282-8050-4d86-8ab1-b2823b7fa1a5-config-out\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.232072 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.232011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.232072 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.232030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.232072 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.232048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.232262 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.232070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.232262 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.232101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.232262 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.232120    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333217 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333170    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333217 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333221    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333439 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333243    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333439 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-config\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333439 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333439 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5dw7\" (UniqueName: \"kubernetes.io/projected/bac34282-8050-4d86-8ab1-b2823b7fa1a5-kube-api-access-l5dw7\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333627 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bac34282-8050-4d86-8ab1-b2823b7fa1a5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333627 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333627 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-web-config\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333627 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333627 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333581    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333627 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333607    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333868 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333868 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333656    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333868 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333682    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bac34282-8050-4d86-8ab1-b2823b7fa1a5-config-out\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333868 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333868 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333762    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333868 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333868 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.333868 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.333861    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.334137 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.334057    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.334229 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.334198    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.335518 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.334909    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.335518 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.334979    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.335518 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.335216    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.335518 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.335346    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.335852 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.335831    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.336333 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.336314    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-config\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.336529 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.336505    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.337871 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.336946    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bac34282-8050-4d86-8ab1-b2823b7fa1a5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.337871 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.337306    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.338021 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.337879    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.338191 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.338169    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bac34282-8050-4d86-8ab1-b2823b7fa1a5-config-out\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.338383 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.338359    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.338648 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.338625    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.339067 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.339046    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.339472 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.339452    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-web-config\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.339592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.339574    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bac34282-8050-4d86-8ab1-b2823b7fa1a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.340234 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.340213    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bac34282-8050-4d86-8ab1-b2823b7fa1a5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.341843 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.341827    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5dw7\" (UniqueName: \"kubernetes.io/projected/bac34282-8050-4d86-8ab1-b2823b7fa1a5-kube-api-access-l5dw7\") pod \"prometheus-k8s-0\" (UID: \"bac34282-8050-4d86-8ab1-b2823b7fa1a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.476083 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.476047    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:13.613490 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:13.613472    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 20:03:13.615458 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:03:13.615429    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac34282_8050_4d86_8ab1_b2823b7fa1a5.slice/crio-8aa247710ee2b3309efa64d4e807aad628041d5185b712c25befb86a69f61dac WatchSource:0}: Error finding container 8aa247710ee2b3309efa64d4e807aad628041d5185b712c25befb86a69f61dac: Status 404 returned error can't find the container with id 8aa247710ee2b3309efa64d4e807aad628041d5185b712c25befb86a69f61dac
Jul 16 20:03:14.102318 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:14.102286    2576 generic.go:356] "Generic (PLEG): container finished" podID="bac34282-8050-4d86-8ab1-b2823b7fa1a5" containerID="45fd4189c3d65433d69970ae2bb12b06fc6f02c22129bafa977a9fab7829cc57" exitCode=0
Jul 16 20:03:14.102733 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:14.102372    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac34282-8050-4d86-8ab1-b2823b7fa1a5","Type":"ContainerDied","Data":"45fd4189c3d65433d69970ae2bb12b06fc6f02c22129bafa977a9fab7829cc57"}
Jul 16 20:03:14.102733 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:14.102457    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac34282-8050-4d86-8ab1-b2823b7fa1a5","Type":"ContainerStarted","Data":"8aa247710ee2b3309efa64d4e807aad628041d5185b712c25befb86a69f61dac"}
Jul 16 20:03:14.244271 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:14.244235    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fa4539a-ad13-4607-a5cd-6b799ca5d0f3" path="/var/lib/kubelet/pods/0fa4539a-ad13-4607-a5cd-6b799ca5d0f3/volumes"
Jul 16 20:03:15.110646 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:15.110607    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac34282-8050-4d86-8ab1-b2823b7fa1a5","Type":"ContainerStarted","Data":"31144d18e1e1dbdb7241b48422041727624657c18f85bd380458bf76bcc20e6f"}
Jul 16 20:03:15.111057 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:15.110725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac34282-8050-4d86-8ab1-b2823b7fa1a5","Type":"ContainerStarted","Data":"76463fb77841d80f2e002ca16312c11902a316e4c4f4d5021cac3c1d4eff0c99"}
Jul 16 20:03:15.111057 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:15.110752    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac34282-8050-4d86-8ab1-b2823b7fa1a5","Type":"ContainerStarted","Data":"b90f3a42c7fe15edd60839c69fba2fbc38bc2cc5ac974335507378535e564d87"}
Jul 16 20:03:15.111057 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:15.110776    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac34282-8050-4d86-8ab1-b2823b7fa1a5","Type":"ContainerStarted","Data":"5e4208a584b50464fadc7e44be7438af93297bb060ae1996fe5e22171bb9d61f"}
Jul 16 20:03:15.111057 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:15.110802    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac34282-8050-4d86-8ab1-b2823b7fa1a5","Type":"ContainerStarted","Data":"8a503770e9c52d503a149e867d57e75e5e68817af6155190189f99b15a936e95"}
Jul 16 20:03:15.111057 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:15.110825    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac34282-8050-4d86-8ab1-b2823b7fa1a5","Type":"ContainerStarted","Data":"b9c4213243491a3617725c6321339cdafae7bdbabd98cd5bf76b8363ff3ca2e0"}
Jul 16 20:03:15.140733 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:15.140666    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.12134229 podStartE2EDuration="2.140649747s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.307457ms" isStatefulPod=false podCreationTimestamp="2026-07-16 20:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:03:15.138406209 +0000 UTC m=+173.389026980" watchObservedRunningTime="2026-07-16 20:03:15.140649747 +0000 UTC m=+173.391270506"
Jul 16 20:03:16.187730 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:16.187681    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69cf4fb89f-fxb2x"]
Jul 16 20:03:18.477105 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:18.477073    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:03:41.208113 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.208048    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69cf4fb89f-fxb2x" podUID="67b1a72c-cc4b-4554-8871-c0a857781b75" containerName="console" containerID="cri-o://0977467fd9a1ca6f3a897119574638b040b7fb0f6932a45c9eed6e2a8791bdb3" gracePeriod=15
Jul 16 20:03:41.450616 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.450592    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:03:41.457972 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.457953    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-console-config\") pod \"67b1a72c-cc4b-4554-8871-c0a857781b75\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") "
Jul 16 20:03:41.458050 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.457984    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-service-ca\") pod \"67b1a72c-cc4b-4554-8871-c0a857781b75\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") "
Jul 16 20:03:41.458050 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.458024    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm7bq\" (UniqueName: \"kubernetes.io/projected/67b1a72c-cc4b-4554-8871-c0a857781b75-kube-api-access-wm7bq\") pod \"67b1a72c-cc4b-4554-8871-c0a857781b75\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") "
Jul 16 20:03:41.458205 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.458166    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67b1a72c-cc4b-4554-8871-c0a857781b75-console-serving-cert\") pod \"67b1a72c-cc4b-4554-8871-c0a857781b75\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") "
Jul 16 20:03:41.458273 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.458251    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67b1a72c-cc4b-4554-8871-c0a857781b75-console-oauth-config\") pod \"67b1a72c-cc4b-4554-8871-c0a857781b75\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") "
Jul 16 20:03:41.458330 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.458308    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-trusted-ca-bundle\") pod \"67b1a72c-cc4b-4554-8871-c0a857781b75\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") "
Jul 16 20:03:41.458387 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.458338    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-oauth-serving-cert\") pod \"67b1a72c-cc4b-4554-8871-c0a857781b75\" (UID: \"67b1a72c-cc4b-4554-8871-c0a857781b75\") "
Jul 16 20:03:41.458440 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.458379    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-console-config" (OuterVolumeSpecName: "console-config") pod "67b1a72c-cc4b-4554-8871-c0a857781b75" (UID: "67b1a72c-cc4b-4554-8871-c0a857781b75"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:03:41.458440 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.458393    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-service-ca" (OuterVolumeSpecName: "service-ca") pod "67b1a72c-cc4b-4554-8871-c0a857781b75" (UID: "67b1a72c-cc4b-4554-8871-c0a857781b75"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:03:41.458719 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.458676    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "67b1a72c-cc4b-4554-8871-c0a857781b75" (UID: "67b1a72c-cc4b-4554-8871-c0a857781b75"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:03:41.458719 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.458682    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-console-config\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:41.459030 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.458737    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "67b1a72c-cc4b-4554-8871-c0a857781b75" (UID: "67b1a72c-cc4b-4554-8871-c0a857781b75"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:03:41.459030 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.458759    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-service-ca\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:41.460298 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.460274    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b1a72c-cc4b-4554-8871-c0a857781b75-kube-api-access-wm7bq" (OuterVolumeSpecName: "kube-api-access-wm7bq") pod "67b1a72c-cc4b-4554-8871-c0a857781b75" (UID: "67b1a72c-cc4b-4554-8871-c0a857781b75"). InnerVolumeSpecName "kube-api-access-wm7bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:03:41.460411 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.460317    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b1a72c-cc4b-4554-8871-c0a857781b75-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "67b1a72c-cc4b-4554-8871-c0a857781b75" (UID: "67b1a72c-cc4b-4554-8871-c0a857781b75"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:03:41.460510 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.460487    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b1a72c-cc4b-4554-8871-c0a857781b75-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "67b1a72c-cc4b-4554-8871-c0a857781b75" (UID: "67b1a72c-cc4b-4554-8871-c0a857781b75"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:03:41.559601 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.559573    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67b1a72c-cc4b-4554-8871-c0a857781b75-console-oauth-config\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:41.559601 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.559599    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-trusted-ca-bundle\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:41.559601 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.559609    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67b1a72c-cc4b-4554-8871-c0a857781b75-oauth-serving-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:41.559837 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.559619    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm7bq\" (UniqueName: \"kubernetes.io/projected/67b1a72c-cc4b-4554-8871-c0a857781b75-kube-api-access-wm7bq\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:41.559837 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:41.559629    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67b1a72c-cc4b-4554-8871-c0a857781b75-console-serving-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:03:42.205673 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:42.205633    2576 generic.go:356] "Generic (PLEG): container finished" podID="67b1a72c-cc4b-4554-8871-c0a857781b75" containerID="0977467fd9a1ca6f3a897119574638b040b7fb0f6932a45c9eed6e2a8791bdb3" exitCode=0
Jul 16 20:03:42.205673 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:42.205669    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69cf4fb89f-fxb2x" event={"ID":"67b1a72c-cc4b-4554-8871-c0a857781b75","Type":"ContainerDied","Data":"0977467fd9a1ca6f3a897119574638b040b7fb0f6932a45c9eed6e2a8791bdb3"}
Jul 16 20:03:42.205959 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:42.205720    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69cf4fb89f-fxb2x"
Jul 16 20:03:42.205959 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:42.205783    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69cf4fb89f-fxb2x" event={"ID":"67b1a72c-cc4b-4554-8871-c0a857781b75","Type":"ContainerDied","Data":"2a8449e26318722ba80e513cbfc79a74ecfca1011fe2da1600d5f0621bc6605f"}
Jul 16 20:03:42.205959 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:42.205811    2576 scope.go:117] "RemoveContainer" containerID="0977467fd9a1ca6f3a897119574638b040b7fb0f6932a45c9eed6e2a8791bdb3"
Jul 16 20:03:42.215125 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:42.215003    2576 scope.go:117] "RemoveContainer" containerID="0977467fd9a1ca6f3a897119574638b040b7fb0f6932a45c9eed6e2a8791bdb3"
Jul 16 20:03:42.215321 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:03:42.215280    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0977467fd9a1ca6f3a897119574638b040b7fb0f6932a45c9eed6e2a8791bdb3\": container with ID starting with 0977467fd9a1ca6f3a897119574638b040b7fb0f6932a45c9eed6e2a8791bdb3 not found: ID does not exist" containerID="0977467fd9a1ca6f3a897119574638b040b7fb0f6932a45c9eed6e2a8791bdb3"
Jul 16 20:03:42.215321 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:42.215304    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0977467fd9a1ca6f3a897119574638b040b7fb0f6932a45c9eed6e2a8791bdb3"} err="failed to get container status \"0977467fd9a1ca6f3a897119574638b040b7fb0f6932a45c9eed6e2a8791bdb3\": rpc error: code = NotFound desc = could not find container \"0977467fd9a1ca6f3a897119574638b040b7fb0f6932a45c9eed6e2a8791bdb3\": container with ID starting with 0977467fd9a1ca6f3a897119574638b040b7fb0f6932a45c9eed6e2a8791bdb3 not found: ID does not exist"
Jul 16 20:03:42.227539 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:42.227517    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69cf4fb89f-fxb2x"]
Jul 16 20:03:42.231199 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:42.231179    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69cf4fb89f-fxb2x"]
Jul 16 20:03:42.243592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:03:42.243565    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67b1a72c-cc4b-4554-8871-c0a857781b75" path="/var/lib/kubelet/pods/67b1a72c-cc4b-4554-8871-c0a857781b75/volumes"
Jul 16 20:04:13.476677 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:04:13.476643    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:04:13.492293 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:04:13.492269    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:04:14.328445 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:04:14.328420    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 20:05:22.106563 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:22.106537    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 20:05:42.597396 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.597361    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"]
Jul 16 20:05:42.598795 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.597612    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b1a72c-cc4b-4554-8871-c0a857781b75" containerName="console"
Jul 16 20:05:42.598795 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.597621    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b1a72c-cc4b-4554-8871-c0a857781b75" containerName="console"
Jul 16 20:05:42.598795 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.597678    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b1a72c-cc4b-4554-8871-c0a857781b75" containerName="console"
Jul 16 20:05:42.599232 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.599216    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:42.604755 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.602301    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 20:05:42.604755 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.603395    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 20:05:42.604755 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.603646    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 16 20:05:42.604755 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.603920    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-sl5q7\""
Jul 16 20:05:42.604755 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.604176    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 16 20:05:42.604755 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.604432    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 16 20:05:42.613382 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.613360    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"]
Jul 16 20:05:42.741442 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.741406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/43802dab-ca6b-4172-ad93-cbbeff6cff69-metrics-cert\") pod \"lws-controller-manager-7b8db6765-p48sw\" (UID: \"43802dab-ca6b-4172-ad93-cbbeff6cff69\") " pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:42.741442 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.741440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shfl7\" (UniqueName: \"kubernetes.io/projected/43802dab-ca6b-4172-ad93-cbbeff6cff69-kube-api-access-shfl7\") pod \"lws-controller-manager-7b8db6765-p48sw\" (UID: \"43802dab-ca6b-4172-ad93-cbbeff6cff69\") " pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:42.741670 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.741478    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43802dab-ca6b-4172-ad93-cbbeff6cff69-cert\") pod \"lws-controller-manager-7b8db6765-p48sw\" (UID: \"43802dab-ca6b-4172-ad93-cbbeff6cff69\") " pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:42.741670 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.741567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/43802dab-ca6b-4172-ad93-cbbeff6cff69-manager-config\") pod \"lws-controller-manager-7b8db6765-p48sw\" (UID: \"43802dab-ca6b-4172-ad93-cbbeff6cff69\") " pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:42.842189 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.842155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43802dab-ca6b-4172-ad93-cbbeff6cff69-cert\") pod \"lws-controller-manager-7b8db6765-p48sw\" (UID: \"43802dab-ca6b-4172-ad93-cbbeff6cff69\") " pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:42.842363 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.842202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/43802dab-ca6b-4172-ad93-cbbeff6cff69-manager-config\") pod \"lws-controller-manager-7b8db6765-p48sw\" (UID: \"43802dab-ca6b-4172-ad93-cbbeff6cff69\") " pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:42.842363 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.842258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/43802dab-ca6b-4172-ad93-cbbeff6cff69-metrics-cert\") pod \"lws-controller-manager-7b8db6765-p48sw\" (UID: \"43802dab-ca6b-4172-ad93-cbbeff6cff69\") " pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:42.842363 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.842275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shfl7\" (UniqueName: \"kubernetes.io/projected/43802dab-ca6b-4172-ad93-cbbeff6cff69-kube-api-access-shfl7\") pod \"lws-controller-manager-7b8db6765-p48sw\" (UID: \"43802dab-ca6b-4172-ad93-cbbeff6cff69\") " pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:42.843022 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.842999    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/43802dab-ca6b-4172-ad93-cbbeff6cff69-manager-config\") pod \"lws-controller-manager-7b8db6765-p48sw\" (UID: \"43802dab-ca6b-4172-ad93-cbbeff6cff69\") " pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:42.844763 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.844745    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43802dab-ca6b-4172-ad93-cbbeff6cff69-cert\") pod \"lws-controller-manager-7b8db6765-p48sw\" (UID: \"43802dab-ca6b-4172-ad93-cbbeff6cff69\") " pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:42.844816 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.844777    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/43802dab-ca6b-4172-ad93-cbbeff6cff69-metrics-cert\") pod \"lws-controller-manager-7b8db6765-p48sw\" (UID: \"43802dab-ca6b-4172-ad93-cbbeff6cff69\") " pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:42.850794 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.850750    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shfl7\" (UniqueName: \"kubernetes.io/projected/43802dab-ca6b-4172-ad93-cbbeff6cff69-kube-api-access-shfl7\") pod \"lws-controller-manager-7b8db6765-p48sw\" (UID: \"43802dab-ca6b-4172-ad93-cbbeff6cff69\") " pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:42.911387 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:42.911352    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:43.050561 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:43.050534    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"]
Jul 16 20:05:43.052983 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:05:43.052953    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43802dab_ca6b_4172_ad93_cbbeff6cff69.slice/crio-e6087ab43a9905ceb252f2682b3a8066a7fc2a11e98f20ba0a4b0d4119a2285d WatchSource:0}: Error finding container e6087ab43a9905ceb252f2682b3a8066a7fc2a11e98f20ba0a4b0d4119a2285d: Status 404 returned error can't find the container with id e6087ab43a9905ceb252f2682b3a8066a7fc2a11e98f20ba0a4b0d4119a2285d
Jul 16 20:05:43.054837 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:43.054818    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 20:05:43.613124 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:43.613093    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw" event={"ID":"43802dab-ca6b-4172-ad93-cbbeff6cff69","Type":"ContainerStarted","Data":"e6087ab43a9905ceb252f2682b3a8066a7fc2a11e98f20ba0a4b0d4119a2285d"}
Jul 16 20:05:45.506999 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.506975    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"]
Jul 16 20:05:45.509786 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.509771    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.515112 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.515087    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 20:05:45.515223 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.515089    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p5l6r\""
Jul 16 20:05:45.515952 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.515935    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 20:05:45.516236 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.516159    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 20:05:45.516236 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.516175    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 20:05:45.543106 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.543076    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"]
Jul 16 20:05:45.566597 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.566567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d75dd15-733c-45c2-909b-11dcf95d5e07-tmp\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.566755 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.566615    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5d75dd15-733c-45c2-909b-11dcf95d5e07-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.566755 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.566657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd98b\" (UniqueName: \"kubernetes.io/projected/5d75dd15-733c-45c2-909b-11dcf95d5e07-kube-api-access-jd98b\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.566755 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.566688    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d75dd15-733c-45c2-909b-11dcf95d5e07-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.566755 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.566730    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5d75dd15-733c-45c2-909b-11dcf95d5e07-manifests\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.622339 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.622300    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw" event={"ID":"43802dab-ca6b-4172-ad93-cbbeff6cff69","Type":"ContainerStarted","Data":"4ce56a4cd72d5cbf22ca8c687d77ebf70ffee651703d0d197a7b30a2f5cd146d"}
Jul 16 20:05:45.622494 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.622402    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:05:45.641662 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.641612    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw" podStartSLOduration=1.235216415 podStartE2EDuration="3.64159841s" totalImagesPullingTime="2.406381995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:05:45.64135189 +0000 UTC m=+323.891972650" watchObservedRunningTime="2026-07-16 20:05:45.64159841 +0000 UTC m=+323.892219169"
Jul 16 20:05:45.667218 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.667189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d75dd15-733c-45c2-909b-11dcf95d5e07-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.667218 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.667221    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5d75dd15-733c-45c2-909b-11dcf95d5e07-manifests\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.667402 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.667249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d75dd15-733c-45c2-909b-11dcf95d5e07-tmp\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.667402 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.667298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5d75dd15-733c-45c2-909b-11dcf95d5e07-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.667402 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.667338    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd98b\" (UniqueName: \"kubernetes.io/projected/5d75dd15-733c-45c2-909b-11dcf95d5e07-kube-api-access-jd98b\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.667680 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.667649    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5d75dd15-733c-45c2-909b-11dcf95d5e07-manifests\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.669670 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.669637    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d75dd15-733c-45c2-909b-11dcf95d5e07-tmp\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.669970 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.669943    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5d75dd15-733c-45c2-909b-11dcf95d5e07-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.670057 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.670033    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d75dd15-733c-45c2-909b-11dcf95d5e07-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.676067 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.676044    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd98b\" (UniqueName: \"kubernetes.io/projected/5d75dd15-733c-45c2-909b-11dcf95d5e07-kube-api-access-jd98b\") pod \"opendatahub-operator-controller-manager-64c594d8d4-hxtq8\" (UID: \"5d75dd15-733c-45c2-909b-11dcf95d5e07\") " pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.820651 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.820625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:45.955011 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:45.954990    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"]
Jul 16 20:05:45.957190 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:05:45.957169    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d75dd15_733c_45c2_909b_11dcf95d5e07.slice/crio-ffc7b06a8e42fb7692115c113133bb51047554a3a3a1a052232623911dc0760d WatchSource:0}: Error finding container ffc7b06a8e42fb7692115c113133bb51047554a3a3a1a052232623911dc0760d: Status 404 returned error can't find the container with id ffc7b06a8e42fb7692115c113133bb51047554a3a3a1a052232623911dc0760d
Jul 16 20:05:46.632257 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:46.632222    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8" event={"ID":"5d75dd15-733c-45c2-909b-11dcf95d5e07","Type":"ContainerStarted","Data":"ffc7b06a8e42fb7692115c113133bb51047554a3a3a1a052232623911dc0760d"}
Jul 16 20:05:50.650740 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:50.650689    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8" event={"ID":"5d75dd15-733c-45c2-909b-11dcf95d5e07","Type":"ContainerStarted","Data":"474be58272c199dfa757c16a2422eb226670e80b83d137e15a13861ca81e56ef"}
Jul 16 20:05:51.655571 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:51.655538    2576 generic.go:356] "Generic (PLEG): container finished" podID="5d75dd15-733c-45c2-909b-11dcf95d5e07" containerID="474be58272c199dfa757c16a2422eb226670e80b83d137e15a13861ca81e56ef" exitCode=0
Jul 16 20:05:51.655952 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:51.655592    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8" event={"ID":"5d75dd15-733c-45c2-909b-11dcf95d5e07","Type":"ContainerDied","Data":"474be58272c199dfa757c16a2422eb226670e80b83d137e15a13861ca81e56ef"}
Jul 16 20:05:52.662437 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:52.662395    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8" event={"ID":"5d75dd15-733c-45c2-909b-11dcf95d5e07","Type":"ContainerStarted","Data":"e6b9635b2a5b393abcf35457b44e156da07131545e9ee43bb5613db6fa6318d5"}
Jul 16 20:05:52.662891 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:52.662560    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:05:52.685810 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:52.685756    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8" podStartSLOduration=2.560064965 podStartE2EDuration="7.68573732s" totalImagesPullingTime="4.115096028s" totalInitContainerRuntime="1.010576327s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:05:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:05:52.684350018 +0000 UTC m=+330.934970776" watchObservedRunningTime="2026-07-16 20:05:52.68573732 +0000 UTC m=+330.936358074"
Jul 16 20:05:56.634844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:05:56.634809    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7b8db6765-p48sw"
Jul 16 20:06:03.668895 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:03.668866    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64c594d8d4-hxtq8"
Jul 16 20:06:07.607079 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.607044    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"]
Jul 16 20:06:07.618767 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.618744    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"
Jul 16 20:06:07.621138 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.621112    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mc88n\""
Jul 16 20:06:07.623499 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.623475    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 16 20:06:07.623766 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.623750    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 20:06:07.624322 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.624302    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 20:06:07.624419 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.624352    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 16 20:06:07.625154 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.625133    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"]
Jul 16 20:06:07.746667 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.746636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34b1e01f-1b19-428b-b13b-66d575c92f2b-tls-certs\") pod \"kube-auth-proxy-59f964d6d4-k6gpj\" (UID: \"34b1e01f-1b19-428b-b13b-66d575c92f2b\") " pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"
Jul 16 20:06:07.746869 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.746682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7dxf\" (UniqueName: \"kubernetes.io/projected/34b1e01f-1b19-428b-b13b-66d575c92f2b-kube-api-access-f7dxf\") pod \"kube-auth-proxy-59f964d6d4-k6gpj\" (UID: \"34b1e01f-1b19-428b-b13b-66d575c92f2b\") " pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"
Jul 16 20:06:07.746869 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.746739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34b1e01f-1b19-428b-b13b-66d575c92f2b-tmp\") pod \"kube-auth-proxy-59f964d6d4-k6gpj\" (UID: \"34b1e01f-1b19-428b-b13b-66d575c92f2b\") " pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"
Jul 16 20:06:07.848199 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.848156    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34b1e01f-1b19-428b-b13b-66d575c92f2b-tls-certs\") pod \"kube-auth-proxy-59f964d6d4-k6gpj\" (UID: \"34b1e01f-1b19-428b-b13b-66d575c92f2b\") " pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"
Jul 16 20:06:07.848363 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.848215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7dxf\" (UniqueName: \"kubernetes.io/projected/34b1e01f-1b19-428b-b13b-66d575c92f2b-kube-api-access-f7dxf\") pod \"kube-auth-proxy-59f964d6d4-k6gpj\" (UID: \"34b1e01f-1b19-428b-b13b-66d575c92f2b\") " pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"
Jul 16 20:06:07.848363 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.848251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34b1e01f-1b19-428b-b13b-66d575c92f2b-tmp\") pod \"kube-auth-proxy-59f964d6d4-k6gpj\" (UID: \"34b1e01f-1b19-428b-b13b-66d575c92f2b\") " pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"
Jul 16 20:06:07.850644 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.850608    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34b1e01f-1b19-428b-b13b-66d575c92f2b-tmp\") pod \"kube-auth-proxy-59f964d6d4-k6gpj\" (UID: \"34b1e01f-1b19-428b-b13b-66d575c92f2b\") " pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"
Jul 16 20:06:07.850859 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.850840    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34b1e01f-1b19-428b-b13b-66d575c92f2b-tls-certs\") pod \"kube-auth-proxy-59f964d6d4-k6gpj\" (UID: \"34b1e01f-1b19-428b-b13b-66d575c92f2b\") " pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"
Jul 16 20:06:07.866228 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.866156    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7dxf\" (UniqueName: \"kubernetes.io/projected/34b1e01f-1b19-428b-b13b-66d575c92f2b-kube-api-access-f7dxf\") pod \"kube-auth-proxy-59f964d6d4-k6gpj\" (UID: \"34b1e01f-1b19-428b-b13b-66d575c92f2b\") " pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"
Jul 16 20:06:07.930133 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:07.930099    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"
Jul 16 20:06:08.091991 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:08.091959    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj"]
Jul 16 20:06:08.095829 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:06:08.095788    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b1e01f_1b19_428b_b13b_66d575c92f2b.slice/crio-0c7ad5a3790ad991a98840b65a5b0f8e1047f77ae5c9a52d6a4b7ca922a6d9bb WatchSource:0}: Error finding container 0c7ad5a3790ad991a98840b65a5b0f8e1047f77ae5c9a52d6a4b7ca922a6d9bb: Status 404 returned error can't find the container with id 0c7ad5a3790ad991a98840b65a5b0f8e1047f77ae5c9a52d6a4b7ca922a6d9bb
Jul 16 20:06:08.722891 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:08.722848    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj" event={"ID":"34b1e01f-1b19-428b-b13b-66d575c92f2b","Type":"ContainerStarted","Data":"0c7ad5a3790ad991a98840b65a5b0f8e1047f77ae5c9a52d6a4b7ca922a6d9bb"}
Jul 16 20:06:12.740205 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:12.740171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj" event={"ID":"34b1e01f-1b19-428b-b13b-66d575c92f2b","Type":"ContainerStarted","Data":"251470a9e91e3643ef1108afd341a44287df7867017064802f1029fb2063193d"}
Jul 16 20:06:12.759322 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:12.759262    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59f964d6d4-k6gpj" podStartSLOduration=1.8150076450000001 podStartE2EDuration="5.759243374s" totalImagesPullingTime="3.944235729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:06:12.758034359 +0000 UTC m=+351.008655136" watchObservedRunningTime="2026-07-16 20:06:12.759243374 +0000 UTC m=+351.009864134"
Jul 16 20:06:16.984951 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:16.984918    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"]
Jul 16 20:06:16.988443 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:16.988420    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:16.991683 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:16.991656    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-9q8vk\""
Jul 16 20:06:16.991969 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:16.991949    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 16 20:06:17.004643 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.004617    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"]
Jul 16 20:06:17.028650 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.028629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d6c98c-0d9f-44e9-8654-d061efb6c892-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.028792 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.028664    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.028792 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.028684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.028792 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.028733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d6c98c-0d9f-44e9-8654-d061efb6c892-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.028792 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.028759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xctd\" (UniqueName: \"kubernetes.io/projected/91d6c98c-0d9f-44e9-8654-d061efb6c892-kube-api-access-8xctd\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.028792 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.028780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.028792 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.028794    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.029008 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.028904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.029008 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.028939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d6c98c-0d9f-44e9-8654-d061efb6c892-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.129603 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.129567    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.129603 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.129606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.129894 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.129632    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d6c98c-0d9f-44e9-8654-d061efb6c892-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.129894 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.129647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xctd\" (UniqueName: \"kubernetes.io/projected/91d6c98c-0d9f-44e9-8654-d061efb6c892-kube-api-access-8xctd\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.129894 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.129673    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.129894 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.129688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.129894 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.129752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.129894 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.129795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d6c98c-0d9f-44e9-8654-d061efb6c892-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.129894 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.129843    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d6c98c-0d9f-44e9-8654-d061efb6c892-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.130237 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.130050    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.130237 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.130078    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.130237 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.130158    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.130376 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.130255    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.130580 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.130565    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d6c98c-0d9f-44e9-8654-d061efb6c892-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.132149 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.132126    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d6c98c-0d9f-44e9-8654-d061efb6c892-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.132336 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.132318    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d6c98c-0d9f-44e9-8654-d061efb6c892-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.147638 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.147612    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d6c98c-0d9f-44e9-8654-d061efb6c892-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.149787 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.149764    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xctd\" (UniqueName: \"kubernetes.io/projected/91d6c98c-0d9f-44e9-8654-d061efb6c892-kube-api-access-8xctd\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49qgp9\" (UID: \"91d6c98c-0d9f-44e9-8654-d061efb6c892\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.300468 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.300387    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:17.458151 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.458122    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"]
Jul 16 20:06:17.459604 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:06:17.459579    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d6c98c_0d9f_44e9_8654_d061efb6c892.slice/crio-0462b15bede908e3ed96c003eb4e7415d9b0a0483989bf3fa207c20c1582bcb5 WatchSource:0}: Error finding container 0462b15bede908e3ed96c003eb4e7415d9b0a0483989bf3fa207c20c1582bcb5: Status 404 returned error can't find the container with id 0462b15bede908e3ed96c003eb4e7415d9b0a0483989bf3fa207c20c1582bcb5
Jul 16 20:06:17.758291 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:17.758260    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9" event={"ID":"91d6c98c-0d9f-44e9-8654-d061efb6c892","Type":"ContainerStarted","Data":"0462b15bede908e3ed96c003eb4e7415d9b0a0483989bf3fa207c20c1582bcb5"}
Jul 16 20:06:18.935861 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:18.935818    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"]
Jul 16 20:06:18.939407 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:18.939386    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"
Jul 16 20:06:18.947062 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:18.947036    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 16 20:06:18.947529 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:18.947487    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kgvx5\""
Jul 16 20:06:18.975967 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:18.975323    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"]
Jul 16 20:06:19.046755 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:19.046723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq287\" (UniqueName: \"kubernetes.io/projected/105589e3-1b76-4c01-af34-2757720c5814-kube-api-access-lq287\") pod \"llmisvc-controller-manager-777c6bd7d5-rkllx\" (UID: \"105589e3-1b76-4c01-af34-2757720c5814\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"
Jul 16 20:06:19.046949 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:19.046781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/105589e3-1b76-4c01-af34-2757720c5814-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rkllx\" (UID: \"105589e3-1b76-4c01-af34-2757720c5814\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"
Jul 16 20:06:19.147720 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:19.147675    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq287\" (UniqueName: \"kubernetes.io/projected/105589e3-1b76-4c01-af34-2757720c5814-kube-api-access-lq287\") pod \"llmisvc-controller-manager-777c6bd7d5-rkllx\" (UID: \"105589e3-1b76-4c01-af34-2757720c5814\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"
Jul 16 20:06:19.147877 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:19.147759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/105589e3-1b76-4c01-af34-2757720c5814-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rkllx\" (UID: \"105589e3-1b76-4c01-af34-2757720c5814\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"
Jul 16 20:06:19.147877 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:06:19.147870    2576 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 16 20:06:19.147959 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:06:19.147932    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105589e3-1b76-4c01-af34-2757720c5814-cert podName:105589e3-1b76-4c01-af34-2757720c5814 nodeName:}" failed. No retries permitted until 2026-07-16 20:06:19.647911699 +0000 UTC m=+357.898532436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/105589e3-1b76-4c01-af34-2757720c5814-cert") pod "llmisvc-controller-manager-777c6bd7d5-rkllx" (UID: "105589e3-1b76-4c01-af34-2757720c5814") : secret "llmisvc-webhook-server-cert" not found
Jul 16 20:06:19.157198 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:19.157170    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq287\" (UniqueName: \"kubernetes.io/projected/105589e3-1b76-4c01-af34-2757720c5814-kube-api-access-lq287\") pod \"llmisvc-controller-manager-777c6bd7d5-rkllx\" (UID: \"105589e3-1b76-4c01-af34-2757720c5814\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"
Jul 16 20:06:19.660870 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:19.660837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/105589e3-1b76-4c01-af34-2757720c5814-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rkllx\" (UID: \"105589e3-1b76-4c01-af34-2757720c5814\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"
Jul 16 20:06:19.663370 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:19.663350    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/105589e3-1b76-4c01-af34-2757720c5814-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rkllx\" (UID: \"105589e3-1b76-4c01-af34-2757720c5814\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"
Jul 16 20:06:19.853252 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:19.853214    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"
Jul 16 20:06:20.919375 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:20.919306    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:06:20.919953 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:20.919414    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:06:20.919953 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:20.919457    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:06:20.988241 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:20.988214    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"]
Jul 16 20:06:20.990509 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:06:20.990480    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod105589e3_1b76_4c01_af34_2757720c5814.slice/crio-d79ace359ddfc21d0259e189b585d77e6ec9a8359d0555553a0ec38ed84c1f24 WatchSource:0}: Error finding container d79ace359ddfc21d0259e189b585d77e6ec9a8359d0555553a0ec38ed84c1f24: Status 404 returned error can't find the container with id d79ace359ddfc21d0259e189b585d77e6ec9a8359d0555553a0ec38ed84c1f24
Jul 16 20:06:21.779380 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:21.779337    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9" event={"ID":"91d6c98c-0d9f-44e9-8654-d061efb6c892","Type":"ContainerStarted","Data":"0463f659b6bde32b29a84feed1068783b764974634d885315849b8d9a8891e9f"}
Jul 16 20:06:21.780334 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:21.780314    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx" event={"ID":"105589e3-1b76-4c01-af34-2757720c5814","Type":"ContainerStarted","Data":"d79ace359ddfc21d0259e189b585d77e6ec9a8359d0555553a0ec38ed84c1f24"}
Jul 16 20:06:21.824789 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:21.824730    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9" podStartSLOduration=2.367230166 podStartE2EDuration="5.824688328s" totalImagesPullingTime="3.457458162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:06:21.823460695 +0000 UTC m=+360.074081465" watchObservedRunningTime="2026-07-16 20:06:21.824688328 +0000 UTC m=+360.075309088"
Jul 16 20:06:22.301580 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:22.301553    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:22.306208 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:22.306186    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:22.784634 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:22.784601    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:22.785536 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:22.785519    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49qgp9"
Jul 16 20:06:24.793946 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:24.793893    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx" event={"ID":"105589e3-1b76-4c01-af34-2757720c5814","Type":"ContainerStarted","Data":"c0012ed7217c5e2baa00021c1958fe61e5430ef6d757a3b5fea0211f96f024ad"}
Jul 16 20:06:25.797905 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:25.797877    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"
Jul 16 20:06:56.804209 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:56.804176    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx"
Jul 16 20:06:56.822352 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:06:56.822301    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rkllx" podStartSLOduration=35.115067828 podStartE2EDuration="38.822286941s" totalImagesPullingTime="3.707219113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:06:24.809215274 +0000 UTC m=+363.059836033" watchObservedRunningTime="2026-07-16 20:06:56.822286941 +0000 UTC m=+395.072907699"
Jul 16 20:08:58.174053 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.174016    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"]
Jul 16 20:08:58.177269 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.177253    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"
Jul 16 20:08:58.179729 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.179692    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 20:08:58.180607 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.180585    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-slrcx\""
Jul 16 20:08:58.180723 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.180588    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 16 20:08:58.180723 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.180588    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 20:08:58.186977 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.186958    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"]
Jul 16 20:08:58.268568 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.268538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgpm2\" (UniqueName: \"kubernetes.io/projected/d4642dd9-98c5-4f0b-b573-1b91a2bfee10-kube-api-access-pgpm2\") pod \"limitador-limitador-64c94fd64f-hbm4c\" (UID: \"d4642dd9-98c5-4f0b-b573-1b91a2bfee10\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"
Jul 16 20:08:58.268568 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.268573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4642dd9-98c5-4f0b-b573-1b91a2bfee10-config-file\") pod \"limitador-limitador-64c94fd64f-hbm4c\" (UID: \"d4642dd9-98c5-4f0b-b573-1b91a2bfee10\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"
Jul 16 20:08:58.271317 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.271285    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"]
Jul 16 20:08:58.369060 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.369015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpm2\" (UniqueName: \"kubernetes.io/projected/d4642dd9-98c5-4f0b-b573-1b91a2bfee10-kube-api-access-pgpm2\") pod \"limitador-limitador-64c94fd64f-hbm4c\" (UID: \"d4642dd9-98c5-4f0b-b573-1b91a2bfee10\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"
Jul 16 20:08:58.369060 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.369051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4642dd9-98c5-4f0b-b573-1b91a2bfee10-config-file\") pod \"limitador-limitador-64c94fd64f-hbm4c\" (UID: \"d4642dd9-98c5-4f0b-b573-1b91a2bfee10\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"
Jul 16 20:08:58.369631 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.369609    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4642dd9-98c5-4f0b-b573-1b91a2bfee10-config-file\") pod \"limitador-limitador-64c94fd64f-hbm4c\" (UID: \"d4642dd9-98c5-4f0b-b573-1b91a2bfee10\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"
Jul 16 20:08:58.390007 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.389986    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpm2\" (UniqueName: \"kubernetes.io/projected/d4642dd9-98c5-4f0b-b573-1b91a2bfee10-kube-api-access-pgpm2\") pod \"limitador-limitador-64c94fd64f-hbm4c\" (UID: \"d4642dd9-98c5-4f0b-b573-1b91a2bfee10\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"
Jul 16 20:08:58.487833 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.487751    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"
Jul 16 20:08:58.625892 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:58.625870    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"]
Jul 16 20:08:58.628399 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:08:58.628371    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4642dd9_98c5_4f0b_b573_1b91a2bfee10.slice/crio-375a4449019e1b67d81342f68b77e049ff0268b6b4624ac20010eba4544fde0b WatchSource:0}: Error finding container 375a4449019e1b67d81342f68b77e049ff0268b6b4624ac20010eba4544fde0b: Status 404 returned error can't find the container with id 375a4449019e1b67d81342f68b77e049ff0268b6b4624ac20010eba4544fde0b
Jul 16 20:08:59.118554 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:59.118524    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-k96fv"]
Jul 16 20:08:59.122744 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:59.122727    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-k96fv"
Jul 16 20:08:59.125498 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:59.125472    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nh77z\""
Jul 16 20:08:59.140819 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:59.140799    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-k96fv"]
Jul 16 20:08:59.175564 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:59.175536    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlzbh\" (UniqueName: \"kubernetes.io/projected/5e74472a-fa15-4f64-a61e-d3b7392d0878-kube-api-access-hlzbh\") pod \"authorino-58cd955675-k96fv\" (UID: \"5e74472a-fa15-4f64-a61e-d3b7392d0878\") " pod="kuadrant-system/authorino-58cd955675-k96fv"
Jul 16 20:08:59.277089 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:59.277026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlzbh\" (UniqueName: \"kubernetes.io/projected/5e74472a-fa15-4f64-a61e-d3b7392d0878-kube-api-access-hlzbh\") pod \"authorino-58cd955675-k96fv\" (UID: \"5e74472a-fa15-4f64-a61e-d3b7392d0878\") " pod="kuadrant-system/authorino-58cd955675-k96fv"
Jul 16 20:08:59.299444 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:59.299390    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlzbh\" (UniqueName: \"kubernetes.io/projected/5e74472a-fa15-4f64-a61e-d3b7392d0878-kube-api-access-hlzbh\") pod \"authorino-58cd955675-k96fv\" (UID: \"5e74472a-fa15-4f64-a61e-d3b7392d0878\") " pod="kuadrant-system/authorino-58cd955675-k96fv"
Jul 16 20:08:59.418592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:59.418500    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c" event={"ID":"d4642dd9-98c5-4f0b-b573-1b91a2bfee10","Type":"ContainerStarted","Data":"375a4449019e1b67d81342f68b77e049ff0268b6b4624ac20010eba4544fde0b"}
Jul 16 20:08:59.442828 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:59.442796    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-k96fv"
Jul 16 20:08:59.608486 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:08:59.608358    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-k96fv"]
Jul 16 20:08:59.611364 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:08:59.611333    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e74472a_fa15_4f64_a61e_d3b7392d0878.slice/crio-a052b40a5233f23c0d1b1613b7def7166cd5af1ab663a854183a3c12bcb57c7e WatchSource:0}: Error finding container a052b40a5233f23c0d1b1613b7def7166cd5af1ab663a854183a3c12bcb57c7e: Status 404 returned error can't find the container with id a052b40a5233f23c0d1b1613b7def7166cd5af1ab663a854183a3c12bcb57c7e
Jul 16 20:09:00.423967 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:00.423938    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-k96fv" event={"ID":"5e74472a-fa15-4f64-a61e-d3b7392d0878","Type":"ContainerStarted","Data":"a052b40a5233f23c0d1b1613b7def7166cd5af1ab663a854183a3c12bcb57c7e"}
Jul 16 20:09:02.434364 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:02.434290    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c" event={"ID":"d4642dd9-98c5-4f0b-b573-1b91a2bfee10","Type":"ContainerStarted","Data":"50565f799af684d34efa1afaf264814a78afe273f280054eae03e0a1828a1da9"}
Jul 16 20:09:02.434871 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:02.434532    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"
Jul 16 20:09:02.463591 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:02.463530    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c" podStartSLOduration=1.679509792 podStartE2EDuration="4.463510879s" totalImagesPullingTime="2.784001087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:09:02.459683533 +0000 UTC m=+520.710304309" watchObservedRunningTime="2026-07-16 20:09:02.463510879 +0000 UTC m=+520.714131712"
Jul 16 20:09:04.446677 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:04.446642    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-k96fv" event={"ID":"5e74472a-fa15-4f64-a61e-d3b7392d0878","Type":"ContainerStarted","Data":"df7edd8c8c9f79958ad7a669367a7129c0c08d5a10a920d179819329cdcc66fa"}
Jul 16 20:09:04.460063 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:04.460017    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-k96fv" podStartSLOduration=0.838439308 podStartE2EDuration="5.460003295s" totalImagesPullingTime="4.621563987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:09:04.459508032 +0000 UTC m=+522.710128810" watchObservedRunningTime="2026-07-16 20:09:04.460003295 +0000 UTC m=+522.710624053"
Jul 16 20:09:05.646798 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:05.646761    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-k96fv"]
Jul 16 20:09:06.454973 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:06.454914    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-k96fv" podUID="5e74472a-fa15-4f64-a61e-d3b7392d0878" containerName="authorino" containerID="cri-o://df7edd8c8c9f79958ad7a669367a7129c0c08d5a10a920d179819329cdcc66fa" gracePeriod=30
Jul 16 20:09:06.698327 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:06.698306    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-k96fv"
Jul 16 20:09:06.738002 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:06.737921    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlzbh\" (UniqueName: \"kubernetes.io/projected/5e74472a-fa15-4f64-a61e-d3b7392d0878-kube-api-access-hlzbh\") pod \"5e74472a-fa15-4f64-a61e-d3b7392d0878\" (UID: \"5e74472a-fa15-4f64-a61e-d3b7392d0878\") "
Jul 16 20:09:06.740068 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:06.740046    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e74472a-fa15-4f64-a61e-d3b7392d0878-kube-api-access-hlzbh" (OuterVolumeSpecName: "kube-api-access-hlzbh") pod "5e74472a-fa15-4f64-a61e-d3b7392d0878" (UID: "5e74472a-fa15-4f64-a61e-d3b7392d0878"). InnerVolumeSpecName "kube-api-access-hlzbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:09:06.839279 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:06.839257    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlzbh\" (UniqueName: \"kubernetes.io/projected/5e74472a-fa15-4f64-a61e-d3b7392d0878-kube-api-access-hlzbh\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:09:07.459634 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:07.459599    2576 generic.go:356] "Generic (PLEG): container finished" podID="5e74472a-fa15-4f64-a61e-d3b7392d0878" containerID="df7edd8c8c9f79958ad7a669367a7129c0c08d5a10a920d179819329cdcc66fa" exitCode=0
Jul 16 20:09:07.459817 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:07.459639    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-k96fv"
Jul 16 20:09:07.459817 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:07.459692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-k96fv" event={"ID":"5e74472a-fa15-4f64-a61e-d3b7392d0878","Type":"ContainerDied","Data":"df7edd8c8c9f79958ad7a669367a7129c0c08d5a10a920d179819329cdcc66fa"}
Jul 16 20:09:07.459817 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:07.459738    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-k96fv" event={"ID":"5e74472a-fa15-4f64-a61e-d3b7392d0878","Type":"ContainerDied","Data":"a052b40a5233f23c0d1b1613b7def7166cd5af1ab663a854183a3c12bcb57c7e"}
Jul 16 20:09:07.459817 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:07.459754    2576 scope.go:117] "RemoveContainer" containerID="df7edd8c8c9f79958ad7a669367a7129c0c08d5a10a920d179819329cdcc66fa"
Jul 16 20:09:07.468494 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:07.468475    2576 scope.go:117] "RemoveContainer" containerID="df7edd8c8c9f79958ad7a669367a7129c0c08d5a10a920d179819329cdcc66fa"
Jul 16 20:09:07.468782 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:09:07.468756    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7edd8c8c9f79958ad7a669367a7129c0c08d5a10a920d179819329cdcc66fa\": container with ID starting with df7edd8c8c9f79958ad7a669367a7129c0c08d5a10a920d179819329cdcc66fa not found: ID does not exist" containerID="df7edd8c8c9f79958ad7a669367a7129c0c08d5a10a920d179819329cdcc66fa"
Jul 16 20:09:07.468872 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:07.468795    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7edd8c8c9f79958ad7a669367a7129c0c08d5a10a920d179819329cdcc66fa"} err="failed to get container status \"df7edd8c8c9f79958ad7a669367a7129c0c08d5a10a920d179819329cdcc66fa\": rpc error: code = NotFound desc = could not find container \"df7edd8c8c9f79958ad7a669367a7129c0c08d5a10a920d179819329cdcc66fa\": container with ID starting with df7edd8c8c9f79958ad7a669367a7129c0c08d5a10a920d179819329cdcc66fa not found: ID does not exist"
Jul 16 20:09:07.485088 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:07.485062    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-k96fv"]
Jul 16 20:09:07.486305 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:07.486286    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-k96fv"]
Jul 16 20:09:08.244293 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:08.244259    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e74472a-fa15-4f64-a61e-d3b7392d0878" path="/var/lib/kubelet/pods/5e74472a-fa15-4f64-a61e-d3b7392d0878/volumes"
Jul 16 20:09:13.441164 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:13.441093    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"
Jul 16 20:09:13.853000 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:13.852970    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"]
Jul 16 20:09:13.853174 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:13.853151    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c" podUID="d4642dd9-98c5-4f0b-b573-1b91a2bfee10" containerName="limitador" containerID="cri-o://50565f799af684d34efa1afaf264814a78afe273f280054eae03e0a1828a1da9" gracePeriod=30
Jul 16 20:09:14.396068 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.396042    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"
Jul 16 20:09:14.494609 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.494521    2576 generic.go:356] "Generic (PLEG): container finished" podID="d4642dd9-98c5-4f0b-b573-1b91a2bfee10" containerID="50565f799af684d34efa1afaf264814a78afe273f280054eae03e0a1828a1da9" exitCode=0
Jul 16 20:09:14.494609 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.494568    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"
Jul 16 20:09:14.494609 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.494572    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c" event={"ID":"d4642dd9-98c5-4f0b-b573-1b91a2bfee10","Type":"ContainerDied","Data":"50565f799af684d34efa1afaf264814a78afe273f280054eae03e0a1828a1da9"}
Jul 16 20:09:14.495055 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.494656    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hbm4c" event={"ID":"d4642dd9-98c5-4f0b-b573-1b91a2bfee10","Type":"ContainerDied","Data":"375a4449019e1b67d81342f68b77e049ff0268b6b4624ac20010eba4544fde0b"}
Jul 16 20:09:14.495055 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.494685    2576 scope.go:117] "RemoveContainer" containerID="50565f799af684d34efa1afaf264814a78afe273f280054eae03e0a1828a1da9"
Jul 16 20:09:14.502261 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.502230    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4642dd9-98c5-4f0b-b573-1b91a2bfee10-config-file\") pod \"d4642dd9-98c5-4f0b-b573-1b91a2bfee10\" (UID: \"d4642dd9-98c5-4f0b-b573-1b91a2bfee10\") "
Jul 16 20:09:14.502406 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.502389    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgpm2\" (UniqueName: \"kubernetes.io/projected/d4642dd9-98c5-4f0b-b573-1b91a2bfee10-kube-api-access-pgpm2\") pod \"d4642dd9-98c5-4f0b-b573-1b91a2bfee10\" (UID: \"d4642dd9-98c5-4f0b-b573-1b91a2bfee10\") "
Jul 16 20:09:14.502619 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.502596    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4642dd9-98c5-4f0b-b573-1b91a2bfee10-config-file" (OuterVolumeSpecName: "config-file") pod "d4642dd9-98c5-4f0b-b573-1b91a2bfee10" (UID: "d4642dd9-98c5-4f0b-b573-1b91a2bfee10"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:09:14.502908 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.502896    2576 scope.go:117] "RemoveContainer" containerID="50565f799af684d34efa1afaf264814a78afe273f280054eae03e0a1828a1da9"
Jul 16 20:09:14.503325 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:09:14.503305    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50565f799af684d34efa1afaf264814a78afe273f280054eae03e0a1828a1da9\": container with ID starting with 50565f799af684d34efa1afaf264814a78afe273f280054eae03e0a1828a1da9 not found: ID does not exist" containerID="50565f799af684d34efa1afaf264814a78afe273f280054eae03e0a1828a1da9"
Jul 16 20:09:14.503375 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.503333    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50565f799af684d34efa1afaf264814a78afe273f280054eae03e0a1828a1da9"} err="failed to get container status \"50565f799af684d34efa1afaf264814a78afe273f280054eae03e0a1828a1da9\": rpc error: code = NotFound desc = could not find container \"50565f799af684d34efa1afaf264814a78afe273f280054eae03e0a1828a1da9\": container with ID starting with 50565f799af684d34efa1afaf264814a78afe273f280054eae03e0a1828a1da9 not found: ID does not exist"
Jul 16 20:09:14.504429 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.504413    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4642dd9-98c5-4f0b-b573-1b91a2bfee10-kube-api-access-pgpm2" (OuterVolumeSpecName: "kube-api-access-pgpm2") pod "d4642dd9-98c5-4f0b-b573-1b91a2bfee10" (UID: "d4642dd9-98c5-4f0b-b573-1b91a2bfee10"). InnerVolumeSpecName "kube-api-access-pgpm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:09:14.603363 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.603333    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgpm2\" (UniqueName: \"kubernetes.io/projected/d4642dd9-98c5-4f0b-b573-1b91a2bfee10-kube-api-access-pgpm2\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:09:14.603363 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.603360    2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4642dd9-98c5-4f0b-b573-1b91a2bfee10-config-file\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:09:14.815114 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.815083    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"]
Jul 16 20:09:14.819090 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:14.819062    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hbm4c"]
Jul 16 20:09:16.244097 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:16.244061    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4642dd9-98c5-4f0b-b573-1b91a2bfee10" path="/var/lib/kubelet/pods/d4642dd9-98c5-4f0b-b573-1b91a2bfee10/volumes"
Jul 16 20:09:20.123573 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.123546    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wklx4"]
Jul 16 20:09:20.123943 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.123844    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4642dd9-98c5-4f0b-b573-1b91a2bfee10" containerName="limitador"
Jul 16 20:09:20.123943 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.123856    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4642dd9-98c5-4f0b-b573-1b91a2bfee10" containerName="limitador"
Jul 16 20:09:20.123943 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.123875    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e74472a-fa15-4f64-a61e-d3b7392d0878" containerName="authorino"
Jul 16 20:09:20.123943 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.123880    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e74472a-fa15-4f64-a61e-d3b7392d0878" containerName="authorino"
Jul 16 20:09:20.123943 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.123935    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e74472a-fa15-4f64-a61e-d3b7392d0878" containerName="authorino"
Jul 16 20:09:20.123943 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.123944    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4642dd9-98c5-4f0b-b573-1b91a2bfee10" containerName="limitador"
Jul 16 20:09:20.128151 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.128134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wklx4"
Jul 16 20:09:20.131737 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.131720    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bpgxw\""
Jul 16 20:09:20.131873 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.131723    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 20:09:20.132593 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.132574    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 20:09:20.132858 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.132843    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 16 20:09:20.136886 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.136865    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wklx4"]
Jul 16 20:09:20.253953 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.253924    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/184532ec-55fd-488e-8af4-38fc5a710ea5-data\") pod \"postgres-69fb4594cf-wklx4\" (UID: \"184532ec-55fd-488e-8af4-38fc5a710ea5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wklx4"
Jul 16 20:09:20.254116 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.253999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s4lc\" (UniqueName: \"kubernetes.io/projected/184532ec-55fd-488e-8af4-38fc5a710ea5-kube-api-access-9s4lc\") pod \"postgres-69fb4594cf-wklx4\" (UID: \"184532ec-55fd-488e-8af4-38fc5a710ea5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wklx4"
Jul 16 20:09:20.355146 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.355116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/184532ec-55fd-488e-8af4-38fc5a710ea5-data\") pod \"postgres-69fb4594cf-wklx4\" (UID: \"184532ec-55fd-488e-8af4-38fc5a710ea5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wklx4"
Jul 16 20:09:20.355290 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.355168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s4lc\" (UniqueName: \"kubernetes.io/projected/184532ec-55fd-488e-8af4-38fc5a710ea5-kube-api-access-9s4lc\") pod \"postgres-69fb4594cf-wklx4\" (UID: \"184532ec-55fd-488e-8af4-38fc5a710ea5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wklx4"
Jul 16 20:09:20.355491 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.355473    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/184532ec-55fd-488e-8af4-38fc5a710ea5-data\") pod \"postgres-69fb4594cf-wklx4\" (UID: \"184532ec-55fd-488e-8af4-38fc5a710ea5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wklx4"
Jul 16 20:09:20.368379 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.368355    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s4lc\" (UniqueName: \"kubernetes.io/projected/184532ec-55fd-488e-8af4-38fc5a710ea5-kube-api-access-9s4lc\") pod \"postgres-69fb4594cf-wklx4\" (UID: \"184532ec-55fd-488e-8af4-38fc5a710ea5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wklx4"
Jul 16 20:09:20.438448 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.438379    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wklx4"
Jul 16 20:09:20.569401 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:20.569369    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wklx4"]
Jul 16 20:09:20.572254 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:09:20.572226    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184532ec_55fd_488e_8af4_38fc5a710ea5.slice/crio-d11261ab0c74c77d7b47ecbeb155e9a1f6a02322942607c1b921b049f8857679 WatchSource:0}: Error finding container d11261ab0c74c77d7b47ecbeb155e9a1f6a02322942607c1b921b049f8857679: Status 404 returned error can't find the container with id d11261ab0c74c77d7b47ecbeb155e9a1f6a02322942607c1b921b049f8857679
Jul 16 20:09:21.527286 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:21.527236    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wklx4" event={"ID":"184532ec-55fd-488e-8af4-38fc5a710ea5","Type":"ContainerStarted","Data":"d11261ab0c74c77d7b47ecbeb155e9a1f6a02322942607c1b921b049f8857679"}
Jul 16 20:09:26.152590 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:26.152564    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 16 20:09:26.552860 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:26.552753    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wklx4" event={"ID":"184532ec-55fd-488e-8af4-38fc5a710ea5","Type":"ContainerStarted","Data":"46d29030103d497397d4837ebfa6c48bf41afcd77f3dac69ca5cd07b57d8e841"}
Jul 16 20:09:26.553034 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:26.552894    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wklx4"
Jul 16 20:09:26.570412 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:26.570366    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wklx4" podStartSLOduration=0.994685968 podStartE2EDuration="6.570351655s" totalImagesPullingTime="5.575665687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:09:26.569073602 +0000 UTC m=+544.819694387" watchObservedRunningTime="2026-07-16 20:09:26.570351655 +0000 UTC m=+544.820972414"
Jul 16 20:09:32.586402 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:32.586370    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wklx4"
Jul 16 20:09:33.136984 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.136950    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gpttk"]
Jul 16 20:09:33.142819 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.142800    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gpttk"
Jul 16 20:09:33.145805 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.145781    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nh77z\""
Jul 16 20:09:33.146582 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.146475    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 20:09:33.146582 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.146507    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 20:09:33.147462 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.147442    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gpttk"]
Jul 16 20:09:33.265886 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.265856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2nwv\" (UniqueName: \"kubernetes.io/projected/0cd7b793-725e-4949-960f-318316c99817-kube-api-access-v2nwv\") pod \"authorino-5d4b67c446-gpttk\" (UID: \"0cd7b793-725e-4949-960f-318316c99817\") " pod="kuadrant-system/authorino-5d4b67c446-gpttk"
Jul 16 20:09:33.367228 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.367189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2nwv\" (UniqueName: \"kubernetes.io/projected/0cd7b793-725e-4949-960f-318316c99817-kube-api-access-v2nwv\") pod \"authorino-5d4b67c446-gpttk\" (UID: \"0cd7b793-725e-4949-960f-318316c99817\") " pod="kuadrant-system/authorino-5d4b67c446-gpttk"
Jul 16 20:09:33.377473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.377441    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2nwv\" (UniqueName: \"kubernetes.io/projected/0cd7b793-725e-4949-960f-318316c99817-kube-api-access-v2nwv\") pod \"authorino-5d4b67c446-gpttk\" (UID: \"0cd7b793-725e-4949-960f-318316c99817\") " pod="kuadrant-system/authorino-5d4b67c446-gpttk"
Jul 16 20:09:33.400015 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.399937    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gpttk"]
Jul 16 20:09:33.400221 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.400205    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gpttk"
Jul 16 20:09:33.443387 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.443350    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bf9bf5b8d-l2pkk"]
Jul 16 20:09:33.448782 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.448760    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf9bf5b8d-l2pkk"
Jul 16 20:09:33.456268 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.456209    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bf9bf5b8d-l2pkk"]
Jul 16 20:09:33.480081 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.480050    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bf9bf5b8d-l2pkk"]
Jul 16 20:09:33.480379 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:09:33.480355    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-l2qg9], unattached volumes=[], failed to process volumes=[kube-api-access-l2qg9]: context canceled" pod="kuadrant-system/authorino-6bf9bf5b8d-l2pkk" podUID="4e6c351f-03eb-452e-8a58-70b7de882da1"
Jul 16 20:09:33.516031 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.516002    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-856fb676-jwvkd"]
Jul 16 20:09:33.519934 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.519909    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-856fb676-jwvkd"
Jul 16 20:09:33.522563 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.522539    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 16 20:09:33.526363 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.526340    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-856fb676-jwvkd"]
Jul 16 20:09:33.537197 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.537176    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gpttk"]
Jul 16 20:09:33.540056 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:09:33.540030    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cd7b793_725e_4949_960f_318316c99817.slice/crio-40499e5dc02ec91900f6cbf1efeca14e35dee7b9cd4f0486025be3836e92da33 WatchSource:0}: Error finding container 40499e5dc02ec91900f6cbf1efeca14e35dee7b9cd4f0486025be3836e92da33: Status 404 returned error can't find the container with id 40499e5dc02ec91900f6cbf1efeca14e35dee7b9cd4f0486025be3836e92da33
Jul 16 20:09:33.569041 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.569019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2qg9\" (UniqueName: \"kubernetes.io/projected/4e6c351f-03eb-452e-8a58-70b7de882da1-kube-api-access-l2qg9\") pod \"authorino-6bf9bf5b8d-l2pkk\" (UID: \"4e6c351f-03eb-452e-8a58-70b7de882da1\") " pod="kuadrant-system/authorino-6bf9bf5b8d-l2pkk"
Jul 16 20:09:33.583399 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.583372    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gpttk" event={"ID":"0cd7b793-725e-4949-960f-318316c99817","Type":"ContainerStarted","Data":"40499e5dc02ec91900f6cbf1efeca14e35dee7b9cd4f0486025be3836e92da33"}
Jul 16 20:09:33.583399 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.583393    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf9bf5b8d-l2pkk"
Jul 16 20:09:33.590560 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.590539    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf9bf5b8d-l2pkk"
Jul 16 20:09:33.670375 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.670292    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8zhb\" (UniqueName: \"kubernetes.io/projected/3e50b7f5-ad48-4e85-b557-51e4aa2bddf6-kube-api-access-f8zhb\") pod \"authorino-856fb676-jwvkd\" (UID: \"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6\") " pod="kuadrant-system/authorino-856fb676-jwvkd"
Jul 16 20:09:33.670375 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.670332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3e50b7f5-ad48-4e85-b557-51e4aa2bddf6-tls-cert\") pod \"authorino-856fb676-jwvkd\" (UID: \"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6\") " pod="kuadrant-system/authorino-856fb676-jwvkd"
Jul 16 20:09:33.670375 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.670354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2qg9\" (UniqueName: \"kubernetes.io/projected/4e6c351f-03eb-452e-8a58-70b7de882da1-kube-api-access-l2qg9\") pod \"authorino-6bf9bf5b8d-l2pkk\" (UID: \"4e6c351f-03eb-452e-8a58-70b7de882da1\") " pod="kuadrant-system/authorino-6bf9bf5b8d-l2pkk"
Jul 16 20:09:33.679978 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.679950    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2qg9\" (UniqueName: \"kubernetes.io/projected/4e6c351f-03eb-452e-8a58-70b7de882da1-kube-api-access-l2qg9\") pod \"authorino-6bf9bf5b8d-l2pkk\" (UID: \"4e6c351f-03eb-452e-8a58-70b7de882da1\") " pod="kuadrant-system/authorino-6bf9bf5b8d-l2pkk"
Jul 16 20:09:33.771637 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.771598    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2qg9\" (UniqueName: \"kubernetes.io/projected/4e6c351f-03eb-452e-8a58-70b7de882da1-kube-api-access-l2qg9\") pod \"4e6c351f-03eb-452e-8a58-70b7de882da1\" (UID: \"4e6c351f-03eb-452e-8a58-70b7de882da1\") "
Jul 16 20:09:33.771827 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.771809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8zhb\" (UniqueName: \"kubernetes.io/projected/3e50b7f5-ad48-4e85-b557-51e4aa2bddf6-kube-api-access-f8zhb\") pod \"authorino-856fb676-jwvkd\" (UID: \"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6\") " pod="kuadrant-system/authorino-856fb676-jwvkd"
Jul 16 20:09:33.771870 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.771846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3e50b7f5-ad48-4e85-b557-51e4aa2bddf6-tls-cert\") pod \"authorino-856fb676-jwvkd\" (UID: \"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6\") " pod="kuadrant-system/authorino-856fb676-jwvkd"
Jul 16 20:09:33.773659 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.773634    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6c351f-03eb-452e-8a58-70b7de882da1-kube-api-access-l2qg9" (OuterVolumeSpecName: "kube-api-access-l2qg9") pod "4e6c351f-03eb-452e-8a58-70b7de882da1" (UID: "4e6c351f-03eb-452e-8a58-70b7de882da1"). InnerVolumeSpecName "kube-api-access-l2qg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:09:33.774323 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.774306    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3e50b7f5-ad48-4e85-b557-51e4aa2bddf6-tls-cert\") pod \"authorino-856fb676-jwvkd\" (UID: \"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6\") " pod="kuadrant-system/authorino-856fb676-jwvkd"
Jul 16 20:09:33.779913 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.779895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8zhb\" (UniqueName: \"kubernetes.io/projected/3e50b7f5-ad48-4e85-b557-51e4aa2bddf6-kube-api-access-f8zhb\") pod \"authorino-856fb676-jwvkd\" (UID: \"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6\") " pod="kuadrant-system/authorino-856fb676-jwvkd"
Jul 16 20:09:33.835355 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.835325    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-856fb676-jwvkd"
Jul 16 20:09:33.872600 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.872565    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2qg9\" (UniqueName: \"kubernetes.io/projected/4e6c351f-03eb-452e-8a58-70b7de882da1-kube-api-access-l2qg9\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:09:33.962684 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:33.962660    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-856fb676-jwvkd"]
Jul 16 20:09:33.965793 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:09:33.965767    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e50b7f5_ad48_4e85_b557_51e4aa2bddf6.slice/crio-06ef5d2b04dfcc588d638956c9c29b4615b97e7c245271eb04a3611d0296b1b0 WatchSource:0}: Error finding container 06ef5d2b04dfcc588d638956c9c29b4615b97e7c245271eb04a3611d0296b1b0: Status 404 returned error can't find the container with id 06ef5d2b04dfcc588d638956c9c29b4615b97e7c245271eb04a3611d0296b1b0
Jul 16 20:09:34.589025 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:34.588985    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gpttk" event={"ID":"0cd7b793-725e-4949-960f-318316c99817","Type":"ContainerStarted","Data":"a986cc17ceb15b47c64b6738ba755e009e7538ccbcc6f6f9a1a667747d99c02f"}
Jul 16 20:09:34.589213 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:34.589062    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-gpttk" podUID="0cd7b793-725e-4949-960f-318316c99817" containerName="authorino" containerID="cri-o://a986cc17ceb15b47c64b6738ba755e009e7538ccbcc6f6f9a1a667747d99c02f" gracePeriod=30
Jul 16 20:09:34.590403 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:34.590382    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-856fb676-jwvkd" event={"ID":"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6","Type":"ContainerStarted","Data":"06ef5d2b04dfcc588d638956c9c29b4615b97e7c245271eb04a3611d0296b1b0"}
Jul 16 20:09:34.590511 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:34.590452    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf9bf5b8d-l2pkk"
Jul 16 20:09:34.622771 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:34.622716    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-gpttk" podStartSLOduration=1.225606379 podStartE2EDuration="1.622676011s" totalImagesPullingTime="397.069632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:09:34.607873027 +0000 UTC m=+552.858493786" watchObservedRunningTime="2026-07-16 20:09:34.622676011 +0000 UTC m=+552.873296771"
Jul 16 20:09:34.631410 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:34.631381    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bf9bf5b8d-l2pkk"]
Jul 16 20:09:34.642609 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:34.637584    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6bf9bf5b8d-l2pkk"]
Jul 16 20:09:34.839463 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:34.839403    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gpttk"
Jul 16 20:09:34.983356 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:34.983328    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2nwv\" (UniqueName: \"kubernetes.io/projected/0cd7b793-725e-4949-960f-318316c99817-kube-api-access-v2nwv\") pod \"0cd7b793-725e-4949-960f-318316c99817\" (UID: \"0cd7b793-725e-4949-960f-318316c99817\") "
Jul 16 20:09:34.985439 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:34.985414    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd7b793-725e-4949-960f-318316c99817-kube-api-access-v2nwv" (OuterVolumeSpecName: "kube-api-access-v2nwv") pod "0cd7b793-725e-4949-960f-318316c99817" (UID: "0cd7b793-725e-4949-960f-318316c99817"). InnerVolumeSpecName "kube-api-access-v2nwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:09:35.084868 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:35.084833    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2nwv\" (UniqueName: \"kubernetes.io/projected/0cd7b793-725e-4949-960f-318316c99817-kube-api-access-v2nwv\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:09:35.595360 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:35.595324    2576 generic.go:356] "Generic (PLEG): container finished" podID="0cd7b793-725e-4949-960f-318316c99817" containerID="a986cc17ceb15b47c64b6738ba755e009e7538ccbcc6f6f9a1a667747d99c02f" exitCode=0
Jul 16 20:09:35.595499 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:35.595368    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gpttk"
Jul 16 20:09:35.595499 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:35.595413    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gpttk" event={"ID":"0cd7b793-725e-4949-960f-318316c99817","Type":"ContainerDied","Data":"a986cc17ceb15b47c64b6738ba755e009e7538ccbcc6f6f9a1a667747d99c02f"}
Jul 16 20:09:35.595499 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:35.595451    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gpttk" event={"ID":"0cd7b793-725e-4949-960f-318316c99817","Type":"ContainerDied","Data":"40499e5dc02ec91900f6cbf1efeca14e35dee7b9cd4f0486025be3836e92da33"}
Jul 16 20:09:35.595499 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:35.595467    2576 scope.go:117] "RemoveContainer" containerID="a986cc17ceb15b47c64b6738ba755e009e7538ccbcc6f6f9a1a667747d99c02f"
Jul 16 20:09:35.604030 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:35.604013    2576 scope.go:117] "RemoveContainer" containerID="a986cc17ceb15b47c64b6738ba755e009e7538ccbcc6f6f9a1a667747d99c02f"
Jul 16 20:09:35.604264 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:09:35.604247    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a986cc17ceb15b47c64b6738ba755e009e7538ccbcc6f6f9a1a667747d99c02f\": container with ID starting with a986cc17ceb15b47c64b6738ba755e009e7538ccbcc6f6f9a1a667747d99c02f not found: ID does not exist" containerID="a986cc17ceb15b47c64b6738ba755e009e7538ccbcc6f6f9a1a667747d99c02f"
Jul 16 20:09:35.604315 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:35.604273    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a986cc17ceb15b47c64b6738ba755e009e7538ccbcc6f6f9a1a667747d99c02f"} err="failed to get container status \"a986cc17ceb15b47c64b6738ba755e009e7538ccbcc6f6f9a1a667747d99c02f\": rpc error: code = NotFound desc = could not find container \"a986cc17ceb15b47c64b6738ba755e009e7538ccbcc6f6f9a1a667747d99c02f\": container with ID starting with a986cc17ceb15b47c64b6738ba755e009e7538ccbcc6f6f9a1a667747d99c02f not found: ID does not exist"
Jul 16 20:09:35.615302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:35.615283    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gpttk"]
Jul 16 20:09:35.620330 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:35.620312    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gpttk"]
Jul 16 20:09:36.244457 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:36.244427    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cd7b793-725e-4949-960f-318316c99817" path="/var/lib/kubelet/pods/0cd7b793-725e-4949-960f-318316c99817/volumes"
Jul 16 20:09:36.244811 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:36.244737    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e6c351f-03eb-452e-8a58-70b7de882da1" path="/var/lib/kubelet/pods/4e6c351f-03eb-452e-8a58-70b7de882da1/volumes"
Jul 16 20:09:37.606277 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:37.606236    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-856fb676-jwvkd" event={"ID":"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6","Type":"ContainerStarted","Data":"65db4306d9133c8cf8ea745cf8b6b72e1e4457722fa637b9980b020b0b571e4c"}
Jul 16 20:09:37.623940 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:09:37.623879    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-856fb676-jwvkd" podStartSLOduration=1.9278094669999999 podStartE2EDuration="4.623858249s" totalImagesPullingTime="2.696048782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:09:37.622584177 +0000 UTC m=+555.873204937" watchObservedRunningTime="2026-07-16 20:09:37.623858249 +0000 UTC m=+555.874479009"
Jul 16 20:10:52.515022 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.514990    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"]
Jul 16 20:10:52.516556 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.515437    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd7b793-725e-4949-960f-318316c99817" containerName="authorino"
Jul 16 20:10:52.516556 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.515452    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd7b793-725e-4949-960f-318316c99817" containerName="authorino"
Jul 16 20:10:52.516556 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.515515    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd7b793-725e-4949-960f-318316c99817" containerName="authorino"
Jul 16 20:10:52.517357 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.517338    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.520871 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.520850    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5xhqz\""
Jul 16 20:10:52.521098 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.520850    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 16 20:10:52.521186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.520876    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 16 20:10:52.521186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.520913    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 16 20:10:52.527583 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.527558    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"]
Jul 16 20:10:52.538206 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.538180    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/622e6119-a07a-41bc-8077-ab2e252667b6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.538319 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.538214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfp9m\" (UniqueName: \"kubernetes.io/projected/622e6119-a07a-41bc-8077-ab2e252667b6-kube-api-access-mfp9m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.538319 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.538258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/622e6119-a07a-41bc-8077-ab2e252667b6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.538319 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.538284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/622e6119-a07a-41bc-8077-ab2e252667b6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.538319 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.538311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/622e6119-a07a-41bc-8077-ab2e252667b6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.538461 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.538370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/622e6119-a07a-41bc-8077-ab2e252667b6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.639464 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.639425    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/622e6119-a07a-41bc-8077-ab2e252667b6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.639645 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.639471    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/622e6119-a07a-41bc-8077-ab2e252667b6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.639645 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.639506    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/622e6119-a07a-41bc-8077-ab2e252667b6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.639645 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.639528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfp9m\" (UniqueName: \"kubernetes.io/projected/622e6119-a07a-41bc-8077-ab2e252667b6-kube-api-access-mfp9m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.639645 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.639595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/622e6119-a07a-41bc-8077-ab2e252667b6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.639645 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.639635    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/622e6119-a07a-41bc-8077-ab2e252667b6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.639926 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.639903    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/622e6119-a07a-41bc-8077-ab2e252667b6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.639991 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.639940    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/622e6119-a07a-41bc-8077-ab2e252667b6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.639991 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.639969    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/622e6119-a07a-41bc-8077-ab2e252667b6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.641920 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.641898    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/622e6119-a07a-41bc-8077-ab2e252667b6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.642104 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.642088    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/622e6119-a07a-41bc-8077-ab2e252667b6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.648969 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.648950    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfp9m\" (UniqueName: \"kubernetes.io/projected/622e6119-a07a-41bc-8077-ab2e252667b6-kube-api-access-mfp9m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-469cj\" (UID: \"622e6119-a07a-41bc-8077-ab2e252667b6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.829311 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.829285    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:10:52.958346 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.958310    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"]
Jul 16 20:10:52.961824 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:10:52.961797    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622e6119_a07a_41bc_8077_ab2e252667b6.slice/crio-0a46207fb488609bd1b6f923010e94ad18f7db4bb547dbf372c2f62bea7ec0b5 WatchSource:0}: Error finding container 0a46207fb488609bd1b6f923010e94ad18f7db4bb547dbf372c2f62bea7ec0b5: Status 404 returned error can't find the container with id 0a46207fb488609bd1b6f923010e94ad18f7db4bb547dbf372c2f62bea7ec0b5
Jul 16 20:10:52.963223 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:52.963197    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 20:10:53.913613 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:53.913563    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj" event={"ID":"622e6119-a07a-41bc-8077-ab2e252667b6","Type":"ContainerStarted","Data":"0a46207fb488609bd1b6f923010e94ad18f7db4bb547dbf372c2f62bea7ec0b5"}
Jul 16 20:10:54.919485 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:54.919449    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj" event={"ID":"622e6119-a07a-41bc-8077-ab2e252667b6","Type":"ContainerStarted","Data":"e5634df9f5e452ec918ccfad342806ed90915948ba1675e266a16229dbe2e2a5"}
Jul 16 20:10:54.939433 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:10:54.939385    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj" podStartSLOduration=1.76150586 podStartE2EDuration="2.939369958s" totalImagesPullingTime="1.177864098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:10:54.937751953 +0000 UTC m=+633.188372707" watchObservedRunningTime="2026-07-16 20:10:54.939369958 +0000 UTC m=+633.189990716"
Jul 16 20:11:02.829839 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:02.829781    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:11:02.830251 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:02.829874    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:11:02.842139 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:02.842117    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:11:02.967853 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:02.967828    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-469cj"
Jul 16 20:11:18.802422 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:18.802387    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"]
Jul 16 20:11:18.805467 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:18.805447    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:18.808061 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:18.808043    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 16 20:11:18.813378 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:18.813356    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"]
Jul 16 20:11:18.974805 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:18.974766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a007a9-185b-4fae-8b71-65475d381620-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:18.974981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:18.974809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk84m\" (UniqueName: \"kubernetes.io/projected/34a007a9-185b-4fae-8b71-65475d381620-kube-api-access-kk84m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:18.974981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:18.974894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a007a9-185b-4fae-8b71-65475d381620-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:18.974981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:18.974920    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a007a9-185b-4fae-8b71-65475d381620-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:18.974981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:18.974957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a007a9-185b-4fae-8b71-65475d381620-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:18.975145 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:18.974984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a007a9-185b-4fae-8b71-65475d381620-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.075768 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.075742    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a007a9-185b-4fae-8b71-65475d381620-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.075928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.075776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk84m\" (UniqueName: \"kubernetes.io/projected/34a007a9-185b-4fae-8b71-65475d381620-kube-api-access-kk84m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.075928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.075816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a007a9-185b-4fae-8b71-65475d381620-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.075928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.075832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a007a9-185b-4fae-8b71-65475d381620-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.075928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.075862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a007a9-185b-4fae-8b71-65475d381620-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.075928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.075881    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a007a9-185b-4fae-8b71-65475d381620-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.076355 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.076331    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a007a9-185b-4fae-8b71-65475d381620-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.076355 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.076348    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a007a9-185b-4fae-8b71-65475d381620-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.076649 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.076403    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a007a9-185b-4fae-8b71-65475d381620-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.078171 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.078150    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a007a9-185b-4fae-8b71-65475d381620-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.078416 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.078399    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a007a9-185b-4fae-8b71-65475d381620-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.085102 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.085083    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk84m\" (UniqueName: \"kubernetes.io/projected/34a007a9-185b-4fae-8b71-65475d381620-kube-api-access-kk84m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7\" (UID: \"34a007a9-185b-4fae-8b71-65475d381620\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.117027 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.117003    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:19.247444 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:19.247413    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"]
Jul 16 20:11:19.250126 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:11:19.250096    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a007a9_185b_4fae_8b71_65475d381620.slice/crio-029580b2e3856ec81c73c688edebdfad46fa1b3ffc24b36cff31b43428884b09 WatchSource:0}: Error finding container 029580b2e3856ec81c73c688edebdfad46fa1b3ffc24b36cff31b43428884b09: Status 404 returned error can't find the container with id 029580b2e3856ec81c73c688edebdfad46fa1b3ffc24b36cff31b43428884b09
Jul 16 20:11:20.025962 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:20.025930    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7" event={"ID":"34a007a9-185b-4fae-8b71-65475d381620","Type":"ContainerStarted","Data":"dfa8f43ea58ccba08824bbd888ac3f9c0d72332e8a849f9572931321d2132af7"}
Jul 16 20:11:20.026355 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:20.025987    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7" event={"ID":"34a007a9-185b-4fae-8b71-65475d381620","Type":"ContainerStarted","Data":"029580b2e3856ec81c73c688edebdfad46fa1b3ffc24b36cff31b43428884b09"}
Jul 16 20:11:20.041824 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:20.041764    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7" podStartSLOduration=1.836883893 podStartE2EDuration="2.041741059s" totalImagesPullingTime="204.857166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:11:20.041608217 +0000 UTC m=+658.292228967" watchObservedRunningTime="2026-07-16 20:11:20.041741059 +0000 UTC m=+658.292361843"
Jul 16 20:11:29.117185 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:29.117144    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:29.117670 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:29.117223    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:29.129960 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:29.129935    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:30.079809 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:30.079783    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7"
Jul 16 20:11:31.118459 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.118430    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"]
Jul 16 20:11:31.124190 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.124166    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.126928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.126905    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 16 20:11:31.133121 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.133094    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"]
Jul 16 20:11:31.284630 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.284598    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a6aa6f-6049-4dfe-992f-1bec11157428-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.284630 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.284631    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a6aa6f-6049-4dfe-992f-1bec11157428-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.284890 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.284654    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a6aa6f-6049-4dfe-992f-1bec11157428-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.284890 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.284676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a6aa6f-6049-4dfe-992f-1bec11157428-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.284890 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.284824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a6aa6f-6049-4dfe-992f-1bec11157428-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.284890 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.284861    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p98w\" (UniqueName: \"kubernetes.io/projected/69a6aa6f-6049-4dfe-992f-1bec11157428-kube-api-access-6p98w\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.386210 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.386128    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a6aa6f-6049-4dfe-992f-1bec11157428-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.386210 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.386170    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p98w\" (UniqueName: \"kubernetes.io/projected/69a6aa6f-6049-4dfe-992f-1bec11157428-kube-api-access-6p98w\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.386426 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.386236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a6aa6f-6049-4dfe-992f-1bec11157428-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.386426 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.386259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a6aa6f-6049-4dfe-992f-1bec11157428-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.386426 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.386285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a6aa6f-6049-4dfe-992f-1bec11157428-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.386426 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.386309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a6aa6f-6049-4dfe-992f-1bec11157428-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.386656 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.386634    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a6aa6f-6049-4dfe-992f-1bec11157428-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.386748 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.386673    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a6aa6f-6049-4dfe-992f-1bec11157428-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.386807 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.386773    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a6aa6f-6049-4dfe-992f-1bec11157428-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.388563 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.388545    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a6aa6f-6049-4dfe-992f-1bec11157428-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.388869 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.388849    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a6aa6f-6049-4dfe-992f-1bec11157428-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.395676 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.395642    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p98w\" (UniqueName: \"kubernetes.io/projected/69a6aa6f-6049-4dfe-992f-1bec11157428-kube-api-access-6p98w\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lj67r\" (UID: \"69a6aa6f-6049-4dfe-992f-1bec11157428\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.434452 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.434429    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:31.570507 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:31.570482    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"]
Jul 16 20:11:31.572663 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:11:31.572636    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a6aa6f_6049_4dfe_992f_1bec11157428.slice/crio-e53920c5379811554b8a049584284feac68fc9b02cb64ccd88876c46c91f4424 WatchSource:0}: Error finding container e53920c5379811554b8a049584284feac68fc9b02cb64ccd88876c46c91f4424: Status 404 returned error can't find the container with id e53920c5379811554b8a049584284feac68fc9b02cb64ccd88876c46c91f4424
Jul 16 20:11:32.078203 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:32.078171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r" event={"ID":"69a6aa6f-6049-4dfe-992f-1bec11157428","Type":"ContainerStarted","Data":"e53920c5379811554b8a049584284feac68fc9b02cb64ccd88876c46c91f4424"}
Jul 16 20:11:33.083385 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:33.083354    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r" event={"ID":"69a6aa6f-6049-4dfe-992f-1bec11157428","Type":"ContainerStarted","Data":"df91da370b33ff69be8016a471861f607d3f5cb7b2d51a2ba81001d657c05a89"}
Jul 16 20:11:33.103571 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:33.103526    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r" podStartSLOduration=1.552221565 podStartE2EDuration="2.103511076s" totalImagesPullingTime="551.289511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:11:33.102502388 +0000 UTC m=+671.353123147" watchObservedRunningTime="2026-07-16 20:11:33.103511076 +0000 UTC m=+671.354131835"
Jul 16 20:11:41.435627 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:41.435583    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:41.436107 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:41.435661    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:41.448235 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:41.448209    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:11:42.137544 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:11:42.137513    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lj67r"
Jul 16 20:12:22.686233 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:22.686142    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-856fb676-jwvkd"]
Jul 16 20:12:22.686798 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:22.686321    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-856fb676-jwvkd" podUID="3e50b7f5-ad48-4e85-b557-51e4aa2bddf6" containerName="authorino" containerID="cri-o://65db4306d9133c8cf8ea745cf8b6b72e1e4457722fa637b9980b020b0b571e4c" gracePeriod=30
Jul 16 20:12:22.940963 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:22.940901    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-856fb676-jwvkd"
Jul 16 20:12:22.962012 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:22.961917    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3e50b7f5-ad48-4e85-b557-51e4aa2bddf6-tls-cert\") pod \"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6\" (UID: \"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6\") "
Jul 16 20:12:22.962261 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:22.962045    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8zhb\" (UniqueName: \"kubernetes.io/projected/3e50b7f5-ad48-4e85-b557-51e4aa2bddf6-kube-api-access-f8zhb\") pod \"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6\" (UID: \"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6\") "
Jul 16 20:12:22.964853 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:22.964809    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e50b7f5-ad48-4e85-b557-51e4aa2bddf6-kube-api-access-f8zhb" (OuterVolumeSpecName: "kube-api-access-f8zhb") pod "3e50b7f5-ad48-4e85-b557-51e4aa2bddf6" (UID: "3e50b7f5-ad48-4e85-b557-51e4aa2bddf6"). InnerVolumeSpecName "kube-api-access-f8zhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:12:22.974245 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:22.974194    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e50b7f5-ad48-4e85-b557-51e4aa2bddf6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3e50b7f5-ad48-4e85-b557-51e4aa2bddf6" (UID: "3e50b7f5-ad48-4e85-b557-51e4aa2bddf6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:12:23.063082 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:23.063052    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8zhb\" (UniqueName: \"kubernetes.io/projected/3e50b7f5-ad48-4e85-b557-51e4aa2bddf6-kube-api-access-f8zhb\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:12:23.063082 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:23.063079    2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3e50b7f5-ad48-4e85-b557-51e4aa2bddf6-tls-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:12:23.304685 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:23.304591    2576 generic.go:356] "Generic (PLEG): container finished" podID="3e50b7f5-ad48-4e85-b557-51e4aa2bddf6" containerID="65db4306d9133c8cf8ea745cf8b6b72e1e4457722fa637b9980b020b0b571e4c" exitCode=0
Jul 16 20:12:23.304685 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:23.304630    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-856fb676-jwvkd" event={"ID":"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6","Type":"ContainerDied","Data":"65db4306d9133c8cf8ea745cf8b6b72e1e4457722fa637b9980b020b0b571e4c"}
Jul 16 20:12:23.304685 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:23.304641    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-856fb676-jwvkd"
Jul 16 20:12:23.304685 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:23.304660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-856fb676-jwvkd" event={"ID":"3e50b7f5-ad48-4e85-b557-51e4aa2bddf6","Type":"ContainerDied","Data":"06ef5d2b04dfcc588d638956c9c29b4615b97e7c245271eb04a3611d0296b1b0"}
Jul 16 20:12:23.304685 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:23.304676    2576 scope.go:117] "RemoveContainer" containerID="65db4306d9133c8cf8ea745cf8b6b72e1e4457722fa637b9980b020b0b571e4c"
Jul 16 20:12:23.313388 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:23.313363    2576 scope.go:117] "RemoveContainer" containerID="65db4306d9133c8cf8ea745cf8b6b72e1e4457722fa637b9980b020b0b571e4c"
Jul 16 20:12:23.313646 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:12:23.313628    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65db4306d9133c8cf8ea745cf8b6b72e1e4457722fa637b9980b020b0b571e4c\": container with ID starting with 65db4306d9133c8cf8ea745cf8b6b72e1e4457722fa637b9980b020b0b571e4c not found: ID does not exist" containerID="65db4306d9133c8cf8ea745cf8b6b72e1e4457722fa637b9980b020b0b571e4c"
Jul 16 20:12:23.313813 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:23.313660    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65db4306d9133c8cf8ea745cf8b6b72e1e4457722fa637b9980b020b0b571e4c"} err="failed to get container status \"65db4306d9133c8cf8ea745cf8b6b72e1e4457722fa637b9980b020b0b571e4c\": rpc error: code = NotFound desc = could not find container \"65db4306d9133c8cf8ea745cf8b6b72e1e4457722fa637b9980b020b0b571e4c\": container with ID starting with 65db4306d9133c8cf8ea745cf8b6b72e1e4457722fa637b9980b020b0b571e4c not found: ID does not exist"
Jul 16 20:12:23.351953 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:23.351928    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-856fb676-jwvkd"]
Jul 16 20:12:23.359104 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:23.359078    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-856fb676-jwvkd"]
Jul 16 20:12:24.245055 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:12:24.245026    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e50b7f5-ad48-4e85-b557-51e4aa2bddf6" path="/var/lib/kubelet/pods/3e50b7f5-ad48-4e85-b557-51e4aa2bddf6/volumes"
Jul 16 20:15:00.137536 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:00.137506    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737215-7vxd2"]
Jul 16 20:15:00.138065 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:00.137844    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e50b7f5-ad48-4e85-b557-51e4aa2bddf6" containerName="authorino"
Jul 16 20:15:00.138065 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:00.137856    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e50b7f5-ad48-4e85-b557-51e4aa2bddf6" containerName="authorino"
Jul 16 20:15:00.138065 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:00.137913    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e50b7f5-ad48-4e85-b557-51e4aa2bddf6" containerName="authorino"
Jul 16 20:15:00.140824 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:00.140809    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737215-7vxd2"
Jul 16 20:15:00.143168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:00.143147    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnh7\""
Jul 16 20:15:00.146012 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:00.145987    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737215-7vxd2"]
Jul 16 20:15:00.153490 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:00.153468    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737215-7vxd2"
Jul 16 20:15:00.289932 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:00.289910    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737215-7vxd2"]
Jul 16 20:15:00.292163 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:15:00.292139    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba5941e_f749_4b4b_acf5_e726ef6ee4d4.slice/crio-d8a64e6fb246099bf4ae305599514b2fc4e3642aeb0ed6f2036440f8d073feae WatchSource:0}: Error finding container d8a64e6fb246099bf4ae305599514b2fc4e3642aeb0ed6f2036440f8d073feae: Status 404 returned error can't find the container with id d8a64e6fb246099bf4ae305599514b2fc4e3642aeb0ed6f2036440f8d073feae
Jul 16 20:15:00.972502 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:00.972466    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737215-7vxd2" event={"ID":"eba5941e-f749-4b4b-acf5-e726ef6ee4d4","Type":"ContainerStarted","Data":"d8a64e6fb246099bf4ae305599514b2fc4e3642aeb0ed6f2036440f8d073feae"}
Jul 16 20:15:02.982753 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:02.982718    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737215-7vxd2" event={"ID":"eba5941e-f749-4b4b-acf5-e726ef6ee4d4","Type":"ContainerStarted","Data":"a383c0e25a1979ea42fe52eec516a6db26ba202c12f18fbbeea20cf9fca156d2"}
Jul 16 20:15:02.999751 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:02.999688    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737215-7vxd2" podStartSLOduration=0.890333303 podStartE2EDuration="2.999670667s" totalImagesPullingTime="2.109337364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:15:02.997657686 +0000 UTC m=+881.248278445" watchObservedRunningTime="2026-07-16 20:15:02.999670667 +0000 UTC m=+881.250291466"
Jul 16 20:15:03.988805 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:03.988770    2576 generic.go:356] "Generic (PLEG): container finished" podID="eba5941e-f749-4b4b-acf5-e726ef6ee4d4" containerID="a383c0e25a1979ea42fe52eec516a6db26ba202c12f18fbbeea20cf9fca156d2" exitCode=0
Jul 16 20:15:03.989185 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:03.988822    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737215-7vxd2" event={"ID":"eba5941e-f749-4b4b-acf5-e726ef6ee4d4","Type":"ContainerDied","Data":"a383c0e25a1979ea42fe52eec516a6db26ba202c12f18fbbeea20cf9fca156d2"}
Jul 16 20:15:05.126395 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:05.126365    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737215-7vxd2"
Jul 16 20:15:05.999130 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:05.999093    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737215-7vxd2" event={"ID":"eba5941e-f749-4b4b-acf5-e726ef6ee4d4","Type":"ContainerDied","Data":"d8a64e6fb246099bf4ae305599514b2fc4e3642aeb0ed6f2036440f8d073feae"}
Jul 16 20:15:05.999297 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:05.999141    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8a64e6fb246099bf4ae305599514b2fc4e3642aeb0ed6f2036440f8d073feae"
Jul 16 20:15:05.999297 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:15:05.999103    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737215-7vxd2"
Jul 16 20:28:51.514148 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.514116    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"]
Jul 16 20:28:51.516618 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.514459    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba5941e-f749-4b4b-acf5-e726ef6ee4d4" containerName="cleanup"
Jul 16 20:28:51.516618 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.514471    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba5941e-f749-4b4b-acf5-e726ef6ee4d4" containerName="cleanup"
Jul 16 20:28:51.516618 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.514527    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba5941e-f749-4b4b-acf5-e726ef6ee4d4" containerName="cleanup"
Jul 16 20:28:51.517554 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.517534    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.520044 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.520023    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-49aaf8ce-openshift-default-dockercfg-j8nth\""
Jul 16 20:28:51.539267 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.539243    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"]
Jul 16 20:28:51.640758 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.640695    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48l8\" (UniqueName: \"kubernetes.io/projected/f21ef905-258b-424e-98b5-1e1c6481aee0-kube-api-access-w48l8\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.640960 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.640809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-workload-certs\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.640960 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.640832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21ef905-258b-424e-98b5-1e1c6481aee0-istiod-ca-cert\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.640960 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.640850    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-envoy\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.640960 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.640927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-data\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.641131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.640970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-credential-socket\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.641131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.641021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-workload-socket\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.641131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.641054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-podinfo\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.641131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.641083    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-token\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.742088 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.742051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48l8\" (UniqueName: \"kubernetes.io/projected/f21ef905-258b-424e-98b5-1e1c6481aee0-kube-api-access-w48l8\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.742258 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.742111    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-workload-certs\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.742258 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.742133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21ef905-258b-424e-98b5-1e1c6481aee0-istiod-ca-cert\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.742258 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.742159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-envoy\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.742258 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.742189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-data\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.742258 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.742218    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-credential-socket\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.742503 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.742261    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-workload-socket\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.742503 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.742299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-podinfo\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.742503 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.742332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-token\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.742664 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.742551    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-workload-certs\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.742664 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.742641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-data\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.742817 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.742763    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-credential-socket\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.742900 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.742873    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-workload-socket\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.743057 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.743036    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21ef905-258b-424e-98b5-1e1c6481aee0-istiod-ca-cert\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.744738 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.744719    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-envoy\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.744841 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.744825    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-podinfo\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.749949 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.749914    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-token\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.750542 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.750520    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48l8\" (UniqueName: \"kubernetes.io/projected/f21ef905-258b-424e-98b5-1e1c6481aee0-kube-api-access-w48l8\") pod \"e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") " pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.830074 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.830047    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:51.966203 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.966181    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"]
Jul 16 20:28:51.968603 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:28:51.968573    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21ef905_258b_424e_98b5_1e1c6481aee0.slice/crio-5db6e9f76f258d1e46294f9d8b8009c70cd692f772b90d484703d041f61653f4 WatchSource:0}: Error finding container 5db6e9f76f258d1e46294f9d8b8009c70cd692f772b90d484703d041f61653f4: Status 404 returned error can't find the container with id 5db6e9f76f258d1e46294f9d8b8009c70cd692f772b90d484703d041f61653f4
Jul 16 20:28:51.970864 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.970827    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:28:51.970951 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.970910    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:28:51.970992 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:51.970951    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:28:52.451039 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:52.451003    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8" event={"ID":"f21ef905-258b-424e-98b5-1e1c6481aee0","Type":"ContainerStarted","Data":"a7927c325e8beec6f87be81b8db5684ac3ea4f82c2b430898826a739367f7c4b"}
Jul 16 20:28:52.451207 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:52.451053    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8" event={"ID":"f21ef905-258b-424e-98b5-1e1c6481aee0","Type":"ContainerStarted","Data":"5db6e9f76f258d1e46294f9d8b8009c70cd692f772b90d484703d041f61653f4"}
Jul 16 20:28:52.474838 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:52.474785    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8" podStartSLOduration=1.474769558 podStartE2EDuration="1.474769558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:28:52.472528077 +0000 UTC m=+1710.723148836" watchObservedRunningTime="2026-07-16 20:28:52.474769558 +0000 UTC m=+1710.725390369"
Jul 16 20:28:52.830134 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:52.830104    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:53.835459 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:53.835431    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:54.460718 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:54.460673    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:28:54.461595 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:28:54.461580    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:29:05.512164 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:05.512086    2576 generic.go:356] "Generic (PLEG): container finished" podID="f21ef905-258b-424e-98b5-1e1c6481aee0" containerID="a7927c325e8beec6f87be81b8db5684ac3ea4f82c2b430898826a739367f7c4b" exitCode=0
Jul 16 20:29:05.512549 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:05.512161    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8" event={"ID":"f21ef905-258b-424e-98b5-1e1c6481aee0","Type":"ContainerDied","Data":"a7927c325e8beec6f87be81b8db5684ac3ea4f82c2b430898826a739367f7c4b"}
Jul 16 20:29:05.512549 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:05.512536    2576 scope.go:117] "RemoveContainer" containerID="a7927c325e8beec6f87be81b8db5684ac3ea4f82c2b430898826a739367f7c4b"
Jul 16 20:29:05.513561 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:05.513535    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:29:05.513686 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:05.513604    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:29:05.513686 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:05.513637    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:29:06.517768 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:06.517736    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8" event={"ID":"f21ef905-258b-424e-98b5-1e1c6481aee0","Type":"ContainerStarted","Data":"edc2d9328cc3685fb99b51d4fd0bce16c2d7bc61d643a9cbf2a10387af15e59e"}
Jul 16 20:29:06.831138 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:06.831108    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:29:06.832196 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:06.832173    2576 patch_prober.go:28] interesting pod/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 16 20:29:06.832306 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:06.832245    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8" podUID="f21ef905-258b-424e-98b5-1e1c6481aee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 16 20:29:07.830566 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:07.830526    2576 patch_prober.go:28] interesting pod/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 16 20:29:07.830941 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:07.830586    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8" podUID="f21ef905-258b-424e-98b5-1e1c6481aee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 16 20:29:08.831292 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:08.831253    2576 patch_prober.go:28] interesting pod/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 16 20:29:08.831695 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:08.831310    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8" podUID="f21ef905-258b-424e-98b5-1e1c6481aee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 16 20:29:09.460833 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:09.460777    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:29:09.831311 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:09.831282    2576 patch_prober.go:28] interesting pod/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 16 20:29:09.831681 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:09.831332    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8" podUID="f21ef905-258b-424e-98b5-1e1c6481aee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 16 20:29:10.834757 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:10.834728    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:29:10.835734 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:10.835715    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:29:42.923092 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:42.923058    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"]
Jul 16 20:29:42.923623 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:42.923246    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8" podUID="f21ef905-258b-424e-98b5-1e1c6481aee0" containerName="istio-proxy" containerID="cri-o://edc2d9328cc3685fb99b51d4fd0bce16c2d7bc61d643a9cbf2a10387af15e59e" gracePeriod=30
Jul 16 20:29:43.578308 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.578273    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"]
Jul 16 20:29:43.581472 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.581457    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.583839 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.583823    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fa642852-gw-openshift-default-dockercfg-dpg4h\""
Jul 16 20:29:43.596939 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.596915    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"]
Jul 16 20:29:43.716870 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.716841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-token\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.717029 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.716884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebfe13f-606c-4a7b-a41b-913e3e253951-istiod-ca-cert\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.717029 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.716908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-credential-socket\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.717029 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.716994    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-data\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.717029 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.717022    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv5qf\" (UniqueName: \"kubernetes.io/projected/6ebfe13f-606c-4a7b-a41b-913e3e253951-kube-api-access-xv5qf\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.717166 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.717043    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-workload-socket\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.717166 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.717123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-workload-certs\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.717166 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.717151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-podinfo\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.717260 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.717173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-envoy\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.818109 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-token\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.818109 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818111    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebfe13f-606c-4a7b-a41b-913e3e253951-istiod-ca-cert\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.818351 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-credential-socket\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.818351 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-data\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.818351 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv5qf\" (UniqueName: \"kubernetes.io/projected/6ebfe13f-606c-4a7b-a41b-913e3e253951-kube-api-access-xv5qf\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.818351 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-workload-socket\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.818351 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-workload-certs\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.818351 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818256    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-podinfo\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.818351 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-envoy\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.818682 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818599    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-data\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.818783 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818751    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-workload-socket\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.818933 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818907    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-credential-socket\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.818933 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818922    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebfe13f-606c-4a7b-a41b-913e3e253951-istiod-ca-cert\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.819077 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.818906    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-workload-certs\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.820849 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.820833    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-envoy\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.821191 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.821170    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-podinfo\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.827479 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.827428    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-token\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.828235 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.828213    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv5qf\" (UniqueName: \"kubernetes.io/projected/6ebfe13f-606c-4a7b-a41b-913e3e253951-kube-api-access-xv5qf\") pod \"e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") " pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:43.892164 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:43.892134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:44.042568 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:44.042537    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"]
Jul 16 20:29:44.046307 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:29:44.046278    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebfe13f_606c_4a7b_a41b_913e3e253951.slice/crio-18de48b491e76897062c1bd0d247a81a97df2dac4ef5e5370c4a0c98bc1158b4 WatchSource:0}: Error finding container 18de48b491e76897062c1bd0d247a81a97df2dac4ef5e5370c4a0c98bc1158b4: Status 404 returned error can't find the container with id 18de48b491e76897062c1bd0d247a81a97df2dac4ef5e5370c4a0c98bc1158b4
Jul 16 20:29:44.048554 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:44.048522    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:29:44.048632 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:44.048586    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:29:44.048632 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:44.048615    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:29:44.689030 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:44.688993    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" event={"ID":"6ebfe13f-606c-4a7b-a41b-913e3e253951","Type":"ContainerStarted","Data":"5d58200e11e9e69b92ed0ebf5a77baac88724b8cc5afbd82cb01c9d2dd4e3116"}
Jul 16 20:29:44.689216 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:44.689065    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" event={"ID":"6ebfe13f-606c-4a7b-a41b-913e3e253951","Type":"ContainerStarted","Data":"18de48b491e76897062c1bd0d247a81a97df2dac4ef5e5370c4a0c98bc1158b4"}
Jul 16 20:29:44.710673 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:44.710621    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" podStartSLOduration=1.710605337 podStartE2EDuration="1.710605337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:29:44.708755932 +0000 UTC m=+1762.959376710" watchObservedRunningTime="2026-07-16 20:29:44.710605337 +0000 UTC m=+1762.961226095"
Jul 16 20:29:44.892721 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:44.892664    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:45.898462 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:45.898431    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:46.696511 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:46.696482    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:46.697499 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:46.697477    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:29:48.178922 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.178900    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:29:48.251807 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.251773    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-workload-socket\") pod \"f21ef905-258b-424e-98b5-1e1c6481aee0\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") "
Jul 16 20:29:48.251981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.251825    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-podinfo\") pod \"f21ef905-258b-424e-98b5-1e1c6481aee0\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") "
Jul 16 20:29:48.251981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.251851    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-workload-certs\") pod \"f21ef905-258b-424e-98b5-1e1c6481aee0\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") "
Jul 16 20:29:48.251981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.251878    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-credential-socket\") pod \"f21ef905-258b-424e-98b5-1e1c6481aee0\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") "
Jul 16 20:29:48.251981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.251909    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w48l8\" (UniqueName: \"kubernetes.io/projected/f21ef905-258b-424e-98b5-1e1c6481aee0-kube-api-access-w48l8\") pod \"f21ef905-258b-424e-98b5-1e1c6481aee0\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") "
Jul 16 20:29:48.251981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.251967    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-envoy\") pod \"f21ef905-258b-424e-98b5-1e1c6481aee0\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") "
Jul 16 20:29:48.252234 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.252002    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21ef905-258b-424e-98b5-1e1c6481aee0-istiod-ca-cert\") pod \"f21ef905-258b-424e-98b5-1e1c6481aee0\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") "
Jul 16 20:29:48.252234 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.252029    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-token\") pod \"f21ef905-258b-424e-98b5-1e1c6481aee0\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") "
Jul 16 20:29:48.252234 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.252081    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-data\") pod \"f21ef905-258b-424e-98b5-1e1c6481aee0\" (UID: \"f21ef905-258b-424e-98b5-1e1c6481aee0\") "
Jul 16 20:29:48.252234 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.252085    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f21ef905-258b-424e-98b5-1e1c6481aee0" (UID: "f21ef905-258b-424e-98b5-1e1c6481aee0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:29:48.252439 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.252247    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f21ef905-258b-424e-98b5-1e1c6481aee0" (UID: "f21ef905-258b-424e-98b5-1e1c6481aee0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:29:48.252516 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.252495    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-workload-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:29:48.252576 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.252525    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-credential-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:29:48.252576 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.252495    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21ef905-258b-424e-98b5-1e1c6481aee0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f21ef905-258b-424e-98b5-1e1c6481aee0" (UID: "f21ef905-258b-424e-98b5-1e1c6481aee0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:29:48.252751 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.252717    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f21ef905-258b-424e-98b5-1e1c6481aee0" (UID: "f21ef905-258b-424e-98b5-1e1c6481aee0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:29:48.253081 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.252994    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-data" (OuterVolumeSpecName: "istio-data") pod "f21ef905-258b-424e-98b5-1e1c6481aee0" (UID: "f21ef905-258b-424e-98b5-1e1c6481aee0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:29:48.254783 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.254734    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f21ef905-258b-424e-98b5-1e1c6481aee0" (UID: "f21ef905-258b-424e-98b5-1e1c6481aee0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:29:48.254899 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.254805    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f21ef905-258b-424e-98b5-1e1c6481aee0" (UID: "f21ef905-258b-424e-98b5-1e1c6481aee0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 20:29:48.254965 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.254940    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-token" (OuterVolumeSpecName: "istio-token") pod "f21ef905-258b-424e-98b5-1e1c6481aee0" (UID: "f21ef905-258b-424e-98b5-1e1c6481aee0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:29:48.255050 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.255029    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21ef905-258b-424e-98b5-1e1c6481aee0-kube-api-access-w48l8" (OuterVolumeSpecName: "kube-api-access-w48l8") pod "f21ef905-258b-424e-98b5-1e1c6481aee0" (UID: "f21ef905-258b-424e-98b5-1e1c6481aee0"). InnerVolumeSpecName "kube-api-access-w48l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:29:48.353136 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.353097    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-data\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:29:48.353136 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.353129    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-podinfo\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:29:48.353136 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.353138    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-workload-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:29:48.353375 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.353149    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w48l8\" (UniqueName: \"kubernetes.io/projected/f21ef905-258b-424e-98b5-1e1c6481aee0-kube-api-access-w48l8\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:29:48.353375 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.353159    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-envoy\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:29:48.353375 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.353168    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21ef905-258b-424e-98b5-1e1c6481aee0-istiod-ca-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:29:48.353375 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.353175    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f21ef905-258b-424e-98b5-1e1c6481aee0-istio-token\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:29:48.707522 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.707489    2576 generic.go:356] "Generic (PLEG): container finished" podID="f21ef905-258b-424e-98b5-1e1c6481aee0" containerID="edc2d9328cc3685fb99b51d4fd0bce16c2d7bc61d643a9cbf2a10387af15e59e" exitCode=0
Jul 16 20:29:48.707689 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.707544    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"
Jul 16 20:29:48.707689 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.707570    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8" event={"ID":"f21ef905-258b-424e-98b5-1e1c6481aee0","Type":"ContainerDied","Data":"edc2d9328cc3685fb99b51d4fd0bce16c2d7bc61d643a9cbf2a10387af15e59e"}
Jul 16 20:29:48.707689 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.707621    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8" event={"ID":"f21ef905-258b-424e-98b5-1e1c6481aee0","Type":"ContainerDied","Data":"5db6e9f76f258d1e46294f9d8b8009c70cd692f772b90d484703d041f61653f4"}
Jul 16 20:29:48.707689 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.707645    2576 scope.go:117] "RemoveContainer" containerID="edc2d9328cc3685fb99b51d4fd0bce16c2d7bc61d643a9cbf2a10387af15e59e"
Jul 16 20:29:48.717549 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.717532    2576 scope.go:117] "RemoveContainer" containerID="a7927c325e8beec6f87be81b8db5684ac3ea4f82c2b430898826a739367f7c4b"
Jul 16 20:29:48.725177 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.725162    2576 scope.go:117] "RemoveContainer" containerID="edc2d9328cc3685fb99b51d4fd0bce16c2d7bc61d643a9cbf2a10387af15e59e"
Jul 16 20:29:48.725420 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:29:48.725401    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc2d9328cc3685fb99b51d4fd0bce16c2d7bc61d643a9cbf2a10387af15e59e\": container with ID starting with edc2d9328cc3685fb99b51d4fd0bce16c2d7bc61d643a9cbf2a10387af15e59e not found: ID does not exist" containerID="edc2d9328cc3685fb99b51d4fd0bce16c2d7bc61d643a9cbf2a10387af15e59e"
Jul 16 20:29:48.725458 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.725430    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc2d9328cc3685fb99b51d4fd0bce16c2d7bc61d643a9cbf2a10387af15e59e"} err="failed to get container status \"edc2d9328cc3685fb99b51d4fd0bce16c2d7bc61d643a9cbf2a10387af15e59e\": rpc error: code = NotFound desc = could not find container \"edc2d9328cc3685fb99b51d4fd0bce16c2d7bc61d643a9cbf2a10387af15e59e\": container with ID starting with edc2d9328cc3685fb99b51d4fd0bce16c2d7bc61d643a9cbf2a10387af15e59e not found: ID does not exist"
Jul 16 20:29:48.725458 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.725446    2576 scope.go:117] "RemoveContainer" containerID="a7927c325e8beec6f87be81b8db5684ac3ea4f82c2b430898826a739367f7c4b"
Jul 16 20:29:48.725690 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:29:48.725673    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7927c325e8beec6f87be81b8db5684ac3ea4f82c2b430898826a739367f7c4b\": container with ID starting with a7927c325e8beec6f87be81b8db5684ac3ea4f82c2b430898826a739367f7c4b not found: ID does not exist" containerID="a7927c325e8beec6f87be81b8db5684ac3ea4f82c2b430898826a739367f7c4b"
Jul 16 20:29:48.725776 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.725755    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7927c325e8beec6f87be81b8db5684ac3ea4f82c2b430898826a739367f7c4b"} err="failed to get container status \"a7927c325e8beec6f87be81b8db5684ac3ea4f82c2b430898826a739367f7c4b\": rpc error: code = NotFound desc = could not find container \"a7927c325e8beec6f87be81b8db5684ac3ea4f82c2b430898826a739367f7c4b\": container with ID starting with a7927c325e8beec6f87be81b8db5684ac3ea4f82c2b430898826a739367f7c4b not found: ID does not exist"
Jul 16 20:29:48.730262 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.730243    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"]
Jul 16 20:29:48.734476 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:48.734451    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-49aaf8ce-openshift-default-7b7f6b8b4f-zjdp8"]
Jul 16 20:29:50.244890 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:29:50.244857    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f21ef905-258b-424e-98b5-1e1c6481aee0" path="/var/lib/kubelet/pods/f21ef905-258b-424e-98b5-1e1c6481aee0/volumes"
Jul 16 20:30:01.697788 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:01.697754    2576 patch_prober.go:28] interesting pod/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 16 20:30:01.698186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:01.697806    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 16 20:30:01.767634 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:01.767607    2576 generic.go:356] "Generic (PLEG): container finished" podID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerID="5d58200e11e9e69b92ed0ebf5a77baac88724b8cc5afbd82cb01c9d2dd4e3116" exitCode=0
Jul 16 20:30:01.767803 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:01.767671    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" event={"ID":"6ebfe13f-606c-4a7b-a41b-913e3e253951","Type":"ContainerDied","Data":"5d58200e11e9e69b92ed0ebf5a77baac88724b8cc5afbd82cb01c9d2dd4e3116"}
Jul 16 20:30:01.768060 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:01.768046    2576 scope.go:117] "RemoveContainer" containerID="5d58200e11e9e69b92ed0ebf5a77baac88724b8cc5afbd82cb01c9d2dd4e3116"
Jul 16 20:30:01.769118 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:01.769090    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:30:01.769210 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:01.769157    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:30:01.769210 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:01.769190    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:30:02.777750 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:02.777680    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" event={"ID":"6ebfe13f-606c-4a7b-a41b-913e3e253951","Type":"ContainerStarted","Data":"3ea6efe598ed3b16d637801e5f8194c3137e800e1ffc1ebb7be79bc4f1214439"}
Jul 16 20:30:02.893027 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:02.892996    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:30:02.893339 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:02.893318    2576 patch_prober.go:28] interesting pod/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 16 20:30:02.893384 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:02.893366    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 16 20:30:03.892548 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:03.892509    2576 patch_prober.go:28] interesting pod/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 16 20:30:03.892974 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:03.892582    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 16 20:30:04.892798 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:04.892763    2576 patch_prober.go:28] interesting pod/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 16 20:30:04.893181 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:04.892831    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 16 20:30:05.893276 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:05.893238    2576 patch_prober.go:28] interesting pod/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 16 20:30:05.893662 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:05.893310    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 16 20:30:06.893198 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:06.893155    2576 patch_prober.go:28] interesting pod/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 16 20:30:06.893379 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:06.893272    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 16 20:30:07.897053 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:07.897020    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:30:07.897408 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:07.897302    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:30:07.898049 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:07.898034    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:30:49.422644 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:49.422601    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"]
Jul 16 20:30:49.423130 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:49.422848    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy" containerID="cri-o://3ea6efe598ed3b16d637801e5f8194c3137e800e1ffc1ebb7be79bc4f1214439" gracePeriod=30
Jul 16 20:30:52.899681 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:52.899645    2576 patch_prober.go:28] interesting pod/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 20:30:52.900093 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:52.899712    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 20:30:54.659516 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.659491    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:30:54.748452 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.748419    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-credential-socket\") pod \"6ebfe13f-606c-4a7b-a41b-913e3e253951\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") "
Jul 16 20:30:54.748621 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.748470    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-token\") pod \"6ebfe13f-606c-4a7b-a41b-913e3e253951\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") "
Jul 16 20:30:54.748621 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.748518    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-envoy\") pod \"6ebfe13f-606c-4a7b-a41b-913e3e253951\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") "
Jul 16 20:30:54.748621 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.748549    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv5qf\" (UniqueName: \"kubernetes.io/projected/6ebfe13f-606c-4a7b-a41b-913e3e253951-kube-api-access-xv5qf\") pod \"6ebfe13f-606c-4a7b-a41b-913e3e253951\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") "
Jul 16 20:30:54.748621 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.748583    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-workload-socket\") pod \"6ebfe13f-606c-4a7b-a41b-913e3e253951\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") "
Jul 16 20:30:54.748621 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.748615    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-workload-certs\") pod \"6ebfe13f-606c-4a7b-a41b-913e3e253951\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") "
Jul 16 20:30:54.748949 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.748661    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-podinfo\") pod \"6ebfe13f-606c-4a7b-a41b-913e3e253951\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") "
Jul 16 20:30:54.748949 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.748782    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebfe13f-606c-4a7b-a41b-913e3e253951-istiod-ca-cert\") pod \"6ebfe13f-606c-4a7b-a41b-913e3e253951\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") "
Jul 16 20:30:54.748949 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.748804    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-data\") pod \"6ebfe13f-606c-4a7b-a41b-913e3e253951\" (UID: \"6ebfe13f-606c-4a7b-a41b-913e3e253951\") "
Jul 16 20:30:54.748949 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.748856    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ebfe13f-606c-4a7b-a41b-913e3e253951" (UID: "6ebfe13f-606c-4a7b-a41b-913e3e253951"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:30:54.749232 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.749010    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ebfe13f-606c-4a7b-a41b-913e3e253951" (UID: "6ebfe13f-606c-4a7b-a41b-913e3e253951"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:30:54.749232 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.749128    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-workload-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:30:54.749232 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.749149    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-credential-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:30:54.749388 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.749266    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebfe13f-606c-4a7b-a41b-913e3e253951-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ebfe13f-606c-4a7b-a41b-913e3e253951" (UID: "6ebfe13f-606c-4a7b-a41b-913e3e253951"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:30:54.749895 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.749863    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ebfe13f-606c-4a7b-a41b-913e3e253951" (UID: "6ebfe13f-606c-4a7b-a41b-913e3e253951"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:30:54.750228 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.750202    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ebfe13f-606c-4a7b-a41b-913e3e253951" (UID: "6ebfe13f-606c-4a7b-a41b-913e3e253951"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:30:54.751287 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.751254    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ebfe13f-606c-4a7b-a41b-913e3e253951" (UID: "6ebfe13f-606c-4a7b-a41b-913e3e253951"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:30:54.751512 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.751493    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ebfe13f-606c-4a7b-a41b-913e3e253951" (UID: "6ebfe13f-606c-4a7b-a41b-913e3e253951"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:30:54.751750 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.751670    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ebfe13f-606c-4a7b-a41b-913e3e253951" (UID: "6ebfe13f-606c-4a7b-a41b-913e3e253951"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 20:30:54.751750 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.751677    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebfe13f-606c-4a7b-a41b-913e3e253951-kube-api-access-xv5qf" (OuterVolumeSpecName: "kube-api-access-xv5qf") pod "6ebfe13f-606c-4a7b-a41b-913e3e253951" (UID: "6ebfe13f-606c-4a7b-a41b-913e3e253951"). InnerVolumeSpecName "kube-api-access-xv5qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:30:54.849802 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.849771    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-token\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:30:54.849802 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.849796    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-envoy\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:30:54.849802 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.849806    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv5qf\" (UniqueName: \"kubernetes.io/projected/6ebfe13f-606c-4a7b-a41b-913e3e253951-kube-api-access-xv5qf\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:30:54.850095 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.849815    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-workload-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:30:54.850095 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.849824    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-podinfo\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:30:54.850095 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.849832    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebfe13f-606c-4a7b-a41b-913e3e253951-istiod-ca-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:30:54.850095 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:54.849839    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebfe13f-606c-4a7b-a41b-913e3e253951-istio-data\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:30:55.008066 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:55.007976    2576 generic.go:356] "Generic (PLEG): container finished" podID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerID="3ea6efe598ed3b16d637801e5f8194c3137e800e1ffc1ebb7be79bc4f1214439" exitCode=0
Jul 16 20:30:55.008066 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:55.008031    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" event={"ID":"6ebfe13f-606c-4a7b-a41b-913e3e253951","Type":"ContainerDied","Data":"3ea6efe598ed3b16d637801e5f8194c3137e800e1ffc1ebb7be79bc4f1214439"}
Jul 16 20:30:55.008280 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:55.008065    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"
Jul 16 20:30:55.008280 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:55.008102    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h" event={"ID":"6ebfe13f-606c-4a7b-a41b-913e3e253951","Type":"ContainerDied","Data":"18de48b491e76897062c1bd0d247a81a97df2dac4ef5e5370c4a0c98bc1158b4"}
Jul 16 20:30:55.008280 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:55.008124    2576 scope.go:117] "RemoveContainer" containerID="3ea6efe598ed3b16d637801e5f8194c3137e800e1ffc1ebb7be79bc4f1214439"
Jul 16 20:30:55.017251 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:55.017232    2576 scope.go:117] "RemoveContainer" containerID="5d58200e11e9e69b92ed0ebf5a77baac88724b8cc5afbd82cb01c9d2dd4e3116"
Jul 16 20:30:55.024876 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:55.024856    2576 scope.go:117] "RemoveContainer" containerID="3ea6efe598ed3b16d637801e5f8194c3137e800e1ffc1ebb7be79bc4f1214439"
Jul 16 20:30:55.025146 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:30:55.025126    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea6efe598ed3b16d637801e5f8194c3137e800e1ffc1ebb7be79bc4f1214439\": container with ID starting with 3ea6efe598ed3b16d637801e5f8194c3137e800e1ffc1ebb7be79bc4f1214439 not found: ID does not exist" containerID="3ea6efe598ed3b16d637801e5f8194c3137e800e1ffc1ebb7be79bc4f1214439"
Jul 16 20:30:55.025198 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:55.025156    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea6efe598ed3b16d637801e5f8194c3137e800e1ffc1ebb7be79bc4f1214439"} err="failed to get container status \"3ea6efe598ed3b16d637801e5f8194c3137e800e1ffc1ebb7be79bc4f1214439\": rpc error: code = NotFound desc = could not find container \"3ea6efe598ed3b16d637801e5f8194c3137e800e1ffc1ebb7be79bc4f1214439\": container with ID starting with 3ea6efe598ed3b16d637801e5f8194c3137e800e1ffc1ebb7be79bc4f1214439 not found: ID does not exist"
Jul 16 20:30:55.025198 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:55.025175    2576 scope.go:117] "RemoveContainer" containerID="5d58200e11e9e69b92ed0ebf5a77baac88724b8cc5afbd82cb01c9d2dd4e3116"
Jul 16 20:30:55.025438 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:30:55.025419    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d58200e11e9e69b92ed0ebf5a77baac88724b8cc5afbd82cb01c9d2dd4e3116\": container with ID starting with 5d58200e11e9e69b92ed0ebf5a77baac88724b8cc5afbd82cb01c9d2dd4e3116 not found: ID does not exist" containerID="5d58200e11e9e69b92ed0ebf5a77baac88724b8cc5afbd82cb01c9d2dd4e3116"
Jul 16 20:30:55.025496 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:55.025449    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d58200e11e9e69b92ed0ebf5a77baac88724b8cc5afbd82cb01c9d2dd4e3116"} err="failed to get container status \"5d58200e11e9e69b92ed0ebf5a77baac88724b8cc5afbd82cb01c9d2dd4e3116\": rpc error: code = NotFound desc = could not find container \"5d58200e11e9e69b92ed0ebf5a77baac88724b8cc5afbd82cb01c9d2dd4e3116\": container with ID starting with 5d58200e11e9e69b92ed0ebf5a77baac88724b8cc5afbd82cb01c9d2dd4e3116 not found: ID does not exist"
Jul 16 20:30:55.055662 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:55.055622    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"]
Jul 16 20:30:55.080806 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:55.080776    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fa642852-gw-openshift-default-575fbf6bbc-jc78h"]
Jul 16 20:30:56.244639 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:30:56.244607    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" path="/var/lib/kubelet/pods/6ebfe13f-606c-4a7b-a41b-913e3e253951/volumes"
Jul 16 20:31:01.896479 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:01.896453    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"]
Jul 16 20:31:01.896812 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:01.896798    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f21ef905-258b-424e-98b5-1e1c6481aee0" containerName="istio-proxy"
Jul 16 20:31:01.896856 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:01.896814    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f21ef905-258b-424e-98b5-1e1c6481aee0" containerName="istio-proxy"
Jul 16 20:31:01.896856 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:01.896826    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy"
Jul 16 20:31:01.896856 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:01.896832    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy"
Jul 16 20:31:01.896945 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:01.896887    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f21ef905-258b-424e-98b5-1e1c6481aee0" containerName="istio-proxy"
Jul 16 20:31:01.896945 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:01.896896    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy"
Jul 16 20:31:01.899961 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:01.899940    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:01.902297 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:01.902278    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e9ceb054-openshift-default-dockercfg-vj7bm\""
Jul 16 20:31:01.912343 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:01.912322    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"]
Jul 16 20:31:02.010964 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.010931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-data\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.010964 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.010963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-token\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.011231 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.010985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istiod-ca-cert\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.011231 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.011012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-podinfo\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.011231 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.011094    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-credential-socket\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.011231 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.011132    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-workload-certs\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.011231 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.011207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvf5n\" (UniqueName: \"kubernetes.io/projected/8cf8253b-e921-45e1-b20c-cdcd38127ec9-kube-api-access-qvf5n\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.011441 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.011256    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-workload-socket\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.011441 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.011282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-envoy\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.112314 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.112284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-workload-socket\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.112314 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.112315    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-envoy\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.112611 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.112342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-data\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.112611 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.112477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-token\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.112611 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.112519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istiod-ca-cert\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.112611 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.112562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-podinfo\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.112611 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.112606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-credential-socket\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.112881 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.112636    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-workload-certs\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.112881 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.112660    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-data\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.112881 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.112736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvf5n\" (UniqueName: \"kubernetes.io/projected/8cf8253b-e921-45e1-b20c-cdcd38127ec9-kube-api-access-qvf5n\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.112881 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.112802    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-workload-socket\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.113059 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.113011    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-credential-socket\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.113227 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.113207    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-workload-certs\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.113518 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.113493    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istiod-ca-cert\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.114928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.114896    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-envoy\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.115182 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.115167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-podinfo\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.121183 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.121159    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvf5n\" (UniqueName: \"kubernetes.io/projected/8cf8253b-e921-45e1-b20c-cdcd38127ec9-kube-api-access-qvf5n\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.121265 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.121187    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-token\") pod \"e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") " pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.210519 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.210435    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:02.341806 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.341775    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"]
Jul 16 20:31:02.344293 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:31:02.344265    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf8253b_e921_45e1_b20c_cdcd38127ec9.slice/crio-c0bd634d1c314916693113ee607ebf8f5de4497de3ef7e6c1e6c6e4bf2f7dcfe WatchSource:0}: Error finding container c0bd634d1c314916693113ee607ebf8f5de4497de3ef7e6c1e6c6e4bf2f7dcfe: Status 404 returned error can't find the container with id c0bd634d1c314916693113ee607ebf8f5de4497de3ef7e6c1e6c6e4bf2f7dcfe
Jul 16 20:31:02.346434 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.346397    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:31:02.346492 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.346480    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:31:02.346534 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:02.346520    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:31:03.045294 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:03.045255    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" event={"ID":"8cf8253b-e921-45e1-b20c-cdcd38127ec9","Type":"ContainerStarted","Data":"1220682eaf4c3b09f343b96a6c48600eaa2382431fe66d52bc3380440b49920d"}
Jul 16 20:31:03.045749 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:03.045317    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" event={"ID":"8cf8253b-e921-45e1-b20c-cdcd38127ec9","Type":"ContainerStarted","Data":"c0bd634d1c314916693113ee607ebf8f5de4497de3ef7e6c1e6c6e4bf2f7dcfe"}
Jul 16 20:31:03.072615 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:03.072560    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" podStartSLOduration=2.072545696 podStartE2EDuration="2.072545696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:31:03.0715875 +0000 UTC m=+1841.322208260" watchObservedRunningTime="2026-07-16 20:31:03.072545696 +0000 UTC m=+1841.323166455"
Jul 16 20:31:03.210804 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:03.210773    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:03.215452 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:03.215430    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:04.049679 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:04.049639    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:04.050726 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:04.050681    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:23.148168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:23.148134    2576 generic.go:356] "Generic (PLEG): container finished" podID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerID="1220682eaf4c3b09f343b96a6c48600eaa2382431fe66d52bc3380440b49920d" exitCode=0
Jul 16 20:31:23.148575 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:23.148208    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" event={"ID":"8cf8253b-e921-45e1-b20c-cdcd38127ec9","Type":"ContainerDied","Data":"1220682eaf4c3b09f343b96a6c48600eaa2382431fe66d52bc3380440b49920d"}
Jul 16 20:31:23.148617 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:23.148589    2576 scope.go:117] "RemoveContainer" containerID="1220682eaf4c3b09f343b96a6c48600eaa2382431fe66d52bc3380440b49920d"
Jul 16 20:31:23.149604 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:23.149577    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:31:23.149849 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:23.149659    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:31:23.149849 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:23.149722    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:31:24.155529 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:24.155499    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" event={"ID":"8cf8253b-e921-45e1-b20c-cdcd38127ec9","Type":"ContainerStarted","Data":"a2710cfc2c3b147b692bbc37e834328ce9e216e945e516eebdadc5f0acaa8f17"}
Jul 16 20:31:24.210766 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:24.210733    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:24.211001 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:24.210980    2576 patch_prober.go:28] interesting pod/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 16 20:31:24.211054 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:24.211027    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 16 20:31:25.211050 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:25.211001    2576 patch_prober.go:28] interesting pod/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 16 20:31:25.211414 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:25.211093    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 16 20:31:26.211219 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:26.211183    2576 patch_prober.go:28] interesting pod/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 16 20:31:26.211679 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:26.211237    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 16 20:31:27.211632 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:27.211601    2576 patch_prober.go:28] interesting pod/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 16 20:31:27.212038 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:27.211653    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 16 20:31:28.211281 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:28.211249    2576 patch_prober.go:28] interesting pod/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 16 20:31:28.211472 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:28.211303    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 16 20:31:29.214848 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:29.214816    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:29.215274 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:29.215072    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:31:29.215775 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:31:29.215756    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:32:24.389985 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.388385    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"]
Jul 16 20:32:24.389985 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.388730    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerName="istio-proxy" containerID="cri-o://a2710cfc2c3b147b692bbc37e834328ce9e216e945e516eebdadc5f0acaa8f17" gracePeriod=30
Jul 16 20:32:24.662490 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.662409    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"]
Jul 16 20:32:24.662934 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.662911    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f21ef905-258b-424e-98b5-1e1c6481aee0" containerName="istio-proxy"
Jul 16 20:32:24.662934 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.662933    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f21ef905-258b-424e-98b5-1e1c6481aee0" containerName="istio-proxy"
Jul 16 20:32:24.663113 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.662951    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy"
Jul 16 20:32:24.663113 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.662959    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy"
Jul 16 20:32:24.663216 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.663113    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ebfe13f-606c-4a7b-a41b-913e3e253951" containerName="istio-proxy"
Jul 16 20:32:24.663216 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.663132    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f21ef905-258b-424e-98b5-1e1c6481aee0" containerName="istio-proxy"
Jul 16 20:32:24.665207 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.665186    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.667761 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.667729    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0d61d6f5-openshift-default-dockercfg-6t2sl\""
Jul 16 20:32:24.677120 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.677096    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"]
Jul 16 20:32:24.769320 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.769287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-token\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.769484 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.769330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istiod-ca-cert\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.769484 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.769366    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-envoy\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.769484 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.769391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-workload-certs\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.769484 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.769415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k4tj\" (UniqueName: \"kubernetes.io/projected/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-kube-api-access-8k4tj\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.769632 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.769484    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-credential-socket\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.769632 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.769511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-podinfo\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.769632 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.769605    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-workload-socket\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.769632 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.769627    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-data\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.870213 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.870180    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-workload-socket\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.870213 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.870218    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-data\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.870470 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.870238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-token\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.870470 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.870258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istiod-ca-cert\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.870470 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.870308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-envoy\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.870470 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.870337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-workload-certs\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.870470 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.870361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k4tj\" (UniqueName: \"kubernetes.io/projected/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-kube-api-access-8k4tj\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.870470 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.870415    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-credential-socket\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.870470 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.870440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-podinfo\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.870861 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.870674    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-data\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.870861 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.870751    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-workload-socket\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.870942 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.870859    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-workload-certs\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.870942 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.870905    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-credential-socket\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.871028 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.871004    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istiod-ca-cert\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.872819 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.872799    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-envoy\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.873065 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.873048    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-podinfo\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.879217 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.879184    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-token\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.879504 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.879478    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k4tj\" (UniqueName: \"kubernetes.io/projected/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-kube-api-access-8k4tj\") pod \"e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") " pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:24.976268 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:24.976171    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:25.118528 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:25.118443    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"]
Jul 16 20:32:25.121260 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:32:25.121224    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf1755d_416a_4bc5_92ce_ceac61fde3c5.slice/crio-40ea2295f0648e1934abf6c3151c27f310ac2a5509884c65bf356f8ca306f944 WatchSource:0}: Error finding container 40ea2295f0648e1934abf6c3151c27f310ac2a5509884c65bf356f8ca306f944: Status 404 returned error can't find the container with id 40ea2295f0648e1934abf6c3151c27f310ac2a5509884c65bf356f8ca306f944
Jul 16 20:32:25.123437 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:25.123407    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:32:25.123527 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:25.123480    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:32:25.123589 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:25.123527    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:32:25.440238 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:25.440208    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" event={"ID":"7bf1755d-416a-4bc5-92ce-ceac61fde3c5","Type":"ContainerStarted","Data":"749631f90cc889aa89d5ba73031f6915eaa99e2d0f3133258e8bde3931b656b6"}
Jul 16 20:32:25.440610 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:25.440254    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" event={"ID":"7bf1755d-416a-4bc5-92ce-ceac61fde3c5","Type":"ContainerStarted","Data":"40ea2295f0648e1934abf6c3151c27f310ac2a5509884c65bf356f8ca306f944"}
Jul 16 20:32:25.462961 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:25.462912    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" podStartSLOduration=1.4628962730000001 podStartE2EDuration="1.462896273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:32:25.461121604 +0000 UTC m=+1923.711742373" watchObservedRunningTime="2026-07-16 20:32:25.462896273 +0000 UTC m=+1923.713517032"
Jul 16 20:32:25.976524 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:25.976489    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:26.981370 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:26.981344    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:27.450468 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:27.450435    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:27.451439 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:27.451422    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:29.218116 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.218079    2576 patch_prober.go:28] interesting pod/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 20:32:29.218553 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.218147    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 20:32:29.640315 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.640291    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:32:29.716332 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.716298    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-envoy\") pod \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") "
Jul 16 20:32:29.716507 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.716339    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-token\") pod \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") "
Jul 16 20:32:29.716507 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.716376    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-workload-certs\") pod \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") "
Jul 16 20:32:29.716507 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.716406    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvf5n\" (UniqueName: \"kubernetes.io/projected/8cf8253b-e921-45e1-b20c-cdcd38127ec9-kube-api-access-qvf5n\") pod \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") "
Jul 16 20:32:29.716507 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.716447    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-workload-socket\") pod \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") "
Jul 16 20:32:29.716507 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.716469    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istiod-ca-cert\") pod \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") "
Jul 16 20:32:29.716886 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.716570    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-credential-socket\") pod \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") "
Jul 16 20:32:29.716886 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.716601    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-data\") pod \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") "
Jul 16 20:32:29.716886 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.716625    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-podinfo\") pod \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\" (UID: \"8cf8253b-e921-45e1-b20c-cdcd38127ec9\") "
Jul 16 20:32:29.716886 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.716747    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8cf8253b-e921-45e1-b20c-cdcd38127ec9" (UID: "8cf8253b-e921-45e1-b20c-cdcd38127ec9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:32:29.716886 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.716761    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8cf8253b-e921-45e1-b20c-cdcd38127ec9" (UID: "8cf8253b-e921-45e1-b20c-cdcd38127ec9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:32:29.716886 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.716814    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8cf8253b-e921-45e1-b20c-cdcd38127ec9" (UID: "8cf8253b-e921-45e1-b20c-cdcd38127ec9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:32:29.717186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.716996    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-workload-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:32:29.717186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.717017    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-workload-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:32:29.717186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.717029    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-credential-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:32:29.717186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.717077    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8cf8253b-e921-45e1-b20c-cdcd38127ec9" (UID: "8cf8253b-e921-45e1-b20c-cdcd38127ec9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:32:29.717523 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.717502    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-data" (OuterVolumeSpecName: "istio-data") pod "8cf8253b-e921-45e1-b20c-cdcd38127ec9" (UID: "8cf8253b-e921-45e1-b20c-cdcd38127ec9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:32:29.718960 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.718935    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf8253b-e921-45e1-b20c-cdcd38127ec9-kube-api-access-qvf5n" (OuterVolumeSpecName: "kube-api-access-qvf5n") pod "8cf8253b-e921-45e1-b20c-cdcd38127ec9" (UID: "8cf8253b-e921-45e1-b20c-cdcd38127ec9"). InnerVolumeSpecName "kube-api-access-qvf5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:32:29.719060 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.718994    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-token" (OuterVolumeSpecName: "istio-token") pod "8cf8253b-e921-45e1-b20c-cdcd38127ec9" (UID: "8cf8253b-e921-45e1-b20c-cdcd38127ec9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:32:29.719437 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.719415    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8cf8253b-e921-45e1-b20c-cdcd38127ec9" (UID: "8cf8253b-e921-45e1-b20c-cdcd38127ec9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 20:32:29.719494 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.719415    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8cf8253b-e921-45e1-b20c-cdcd38127ec9" (UID: "8cf8253b-e921-45e1-b20c-cdcd38127ec9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:32:29.817969 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.817939    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvf5n\" (UniqueName: \"kubernetes.io/projected/8cf8253b-e921-45e1-b20c-cdcd38127ec9-kube-api-access-qvf5n\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:32:29.817969 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.817968    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istiod-ca-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:32:29.818150 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.817977    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-data\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:32:29.818150 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.817986    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-podinfo\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:32:29.818150 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.817993    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-envoy\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:32:29.818150 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:29.818001    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf8253b-e921-45e1-b20c-cdcd38127ec9-istio-token\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:32:30.464018 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:30.463986    2576 generic.go:356] "Generic (PLEG): container finished" podID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerID="a2710cfc2c3b147b692bbc37e834328ce9e216e945e516eebdadc5f0acaa8f17" exitCode=0
Jul 16 20:32:30.464456 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:30.464033    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"
Jul 16 20:32:30.464456 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:30.464079    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" event={"ID":"8cf8253b-e921-45e1-b20c-cdcd38127ec9","Type":"ContainerDied","Data":"a2710cfc2c3b147b692bbc37e834328ce9e216e945e516eebdadc5f0acaa8f17"}
Jul 16 20:32:30.464456 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:30.464135    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t" event={"ID":"8cf8253b-e921-45e1-b20c-cdcd38127ec9","Type":"ContainerDied","Data":"c0bd634d1c314916693113ee607ebf8f5de4497de3ef7e6c1e6c6e4bf2f7dcfe"}
Jul 16 20:32:30.464456 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:30.464160    2576 scope.go:117] "RemoveContainer" containerID="a2710cfc2c3b147b692bbc37e834328ce9e216e945e516eebdadc5f0acaa8f17"
Jul 16 20:32:30.472635 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:30.472616    2576 scope.go:117] "RemoveContainer" containerID="1220682eaf4c3b09f343b96a6c48600eaa2382431fe66d52bc3380440b49920d"
Jul 16 20:32:30.480012 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:30.479994    2576 scope.go:117] "RemoveContainer" containerID="a2710cfc2c3b147b692bbc37e834328ce9e216e945e516eebdadc5f0acaa8f17"
Jul 16 20:32:30.480284 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:32:30.480264    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2710cfc2c3b147b692bbc37e834328ce9e216e945e516eebdadc5f0acaa8f17\": container with ID starting with a2710cfc2c3b147b692bbc37e834328ce9e216e945e516eebdadc5f0acaa8f17 not found: ID does not exist" containerID="a2710cfc2c3b147b692bbc37e834328ce9e216e945e516eebdadc5f0acaa8f17"
Jul 16 20:32:30.480328 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:30.480291    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2710cfc2c3b147b692bbc37e834328ce9e216e945e516eebdadc5f0acaa8f17"} err="failed to get container status \"a2710cfc2c3b147b692bbc37e834328ce9e216e945e516eebdadc5f0acaa8f17\": rpc error: code = NotFound desc = could not find container \"a2710cfc2c3b147b692bbc37e834328ce9e216e945e516eebdadc5f0acaa8f17\": container with ID starting with a2710cfc2c3b147b692bbc37e834328ce9e216e945e516eebdadc5f0acaa8f17 not found: ID does not exist"
Jul 16 20:32:30.480328 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:30.480308    2576 scope.go:117] "RemoveContainer" containerID="1220682eaf4c3b09f343b96a6c48600eaa2382431fe66d52bc3380440b49920d"
Jul 16 20:32:30.480552 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:32:30.480533    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1220682eaf4c3b09f343b96a6c48600eaa2382431fe66d52bc3380440b49920d\": container with ID starting with 1220682eaf4c3b09f343b96a6c48600eaa2382431fe66d52bc3380440b49920d not found: ID does not exist" containerID="1220682eaf4c3b09f343b96a6c48600eaa2382431fe66d52bc3380440b49920d"
Jul 16 20:32:30.480598 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:30.480559    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1220682eaf4c3b09f343b96a6c48600eaa2382431fe66d52bc3380440b49920d"} err="failed to get container status \"1220682eaf4c3b09f343b96a6c48600eaa2382431fe66d52bc3380440b49920d\": rpc error: code = NotFound desc = could not find container \"1220682eaf4c3b09f343b96a6c48600eaa2382431fe66d52bc3380440b49920d\": container with ID starting with 1220682eaf4c3b09f343b96a6c48600eaa2382431fe66d52bc3380440b49920d not found: ID does not exist"
Jul 16 20:32:30.486497 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:30.486475    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"]
Jul 16 20:32:30.493459 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:30.493430    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e9ceb054-openshift-default-7f76df9f6f-vsk5t"]
Jul 16 20:32:32.244354 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:32.244282    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" path="/var/lib/kubelet/pods/8cf8253b-e921-45e1-b20c-cdcd38127ec9/volumes"
Jul 16 20:32:43.492875 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:43.492840    2576 patch_prober.go:28] interesting pod/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 16 20:32:43.493280 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:43.492919    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 16 20:32:44.528605 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:44.528519    2576 generic.go:356] "Generic (PLEG): container finished" podID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerID="749631f90cc889aa89d5ba73031f6915eaa99e2d0f3133258e8bde3931b656b6" exitCode=0
Jul 16 20:32:44.528605 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:44.528593    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" event={"ID":"7bf1755d-416a-4bc5-92ce-ceac61fde3c5","Type":"ContainerDied","Data":"749631f90cc889aa89d5ba73031f6915eaa99e2d0f3133258e8bde3931b656b6"}
Jul 16 20:32:44.529094 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:44.528987    2576 scope.go:117] "RemoveContainer" containerID="749631f90cc889aa89d5ba73031f6915eaa99e2d0f3133258e8bde3931b656b6"
Jul 16 20:32:44.529934 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:44.529909    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:32:44.529993 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:44.529975    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:32:44.530038 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:44.530012    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:32:45.535425 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:45.535395    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" event={"ID":"7bf1755d-416a-4bc5-92ce-ceac61fde3c5","Type":"ContainerStarted","Data":"fcd11d7297dac050fa2cbdfa35bda8bd202590a55a3825ffa6bac382398612b1"}
Jul 16 20:32:45.977680 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:45.977640    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:45.977926 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:45.977795    2576 patch_prober.go:28] interesting pod/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 16 20:32:45.977926 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:45.977841    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 16 20:32:46.977300 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:46.977263    2576 patch_prober.go:28] interesting pod/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 16 20:32:46.978351 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:46.978314    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 16 20:32:47.976992 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:47.976955    2576 patch_prober.go:28] interesting pod/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 16 20:32:47.977297 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:47.977267    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 16 20:32:48.977032 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:48.976977    2576 patch_prober.go:28] interesting pod/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 16 20:32:48.977554 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:48.977059    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 16 20:32:49.980745 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:49.980708    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:49.981113 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:49.980979    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:32:49.981737 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:32:49.981716    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:33:45.473268 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:45.473228    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"]
Jul 16 20:33:45.473847 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:45.473492    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerName="istio-proxy" containerID="cri-o://fcd11d7297dac050fa2cbdfa35bda8bd202590a55a3825ffa6bac382398612b1" gracePeriod=30
Jul 16 20:33:49.983053 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:49.983017    2576 patch_prober.go:28] interesting pod/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 20:33:49.983437 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:49.983085    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 20:33:50.720001 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.719973    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:33:50.751078 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.750974    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-data\") pod \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") "
Jul 16 20:33:50.751078 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.751054    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-workload-certs\") pod \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") "
Jul 16 20:33:50.751292 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.751091    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istiod-ca-cert\") pod \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") "
Jul 16 20:33:50.751292 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.751117    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k4tj\" (UniqueName: \"kubernetes.io/projected/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-kube-api-access-8k4tj\") pod \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") "
Jul 16 20:33:50.751292 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.751192    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-envoy\") pod \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") "
Jul 16 20:33:50.751292 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.751231    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-credential-socket\") pod \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") "
Jul 16 20:33:50.751292 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.751281    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-token\") pod \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") "
Jul 16 20:33:50.751539 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.751307    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-podinfo\") pod \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") "
Jul 16 20:33:50.751539 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.751338    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-workload-socket\") pod \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\" (UID: \"7bf1755d-416a-4bc5-92ce-ceac61fde3c5\") "
Jul 16 20:33:50.751539 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.751443    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7bf1755d-416a-4bc5-92ce-ceac61fde3c5" (UID: "7bf1755d-416a-4bc5-92ce-ceac61fde3c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:33:50.751687 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.751629    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istiod-ca-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:33:50.752071 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.751777    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7bf1755d-416a-4bc5-92ce-ceac61fde3c5" (UID: "7bf1755d-416a-4bc5-92ce-ceac61fde3c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:33:50.752071 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.752026    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7bf1755d-416a-4bc5-92ce-ceac61fde3c5" (UID: "7bf1755d-416a-4bc5-92ce-ceac61fde3c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:33:50.752321 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.752297    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7bf1755d-416a-4bc5-92ce-ceac61fde3c5" (UID: "7bf1755d-416a-4bc5-92ce-ceac61fde3c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:33:50.752795 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.752766    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "7bf1755d-416a-4bc5-92ce-ceac61fde3c5" (UID: "7bf1755d-416a-4bc5-92ce-ceac61fde3c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:33:50.754026 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.754005    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7bf1755d-416a-4bc5-92ce-ceac61fde3c5" (UID: "7bf1755d-416a-4bc5-92ce-ceac61fde3c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:33:50.754362 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.754334    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-kube-api-access-8k4tj" (OuterVolumeSpecName: "kube-api-access-8k4tj") pod "7bf1755d-416a-4bc5-92ce-ceac61fde3c5" (UID: "7bf1755d-416a-4bc5-92ce-ceac61fde3c5"). InnerVolumeSpecName "kube-api-access-8k4tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:33:50.754584 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.754568    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "7bf1755d-416a-4bc5-92ce-ceac61fde3c5" (UID: "7bf1755d-416a-4bc5-92ce-ceac61fde3c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:33:50.754748 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.754727    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7bf1755d-416a-4bc5-92ce-ceac61fde3c5" (UID: "7bf1755d-416a-4bc5-92ce-ceac61fde3c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 20:33:50.823800 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.823766    2576 generic.go:356] "Generic (PLEG): container finished" podID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerID="fcd11d7297dac050fa2cbdfa35bda8bd202590a55a3825ffa6bac382398612b1" exitCode=0
Jul 16 20:33:50.823970 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.823811    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"
Jul 16 20:33:50.823970 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.823854    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" event={"ID":"7bf1755d-416a-4bc5-92ce-ceac61fde3c5","Type":"ContainerDied","Data":"fcd11d7297dac050fa2cbdfa35bda8bd202590a55a3825ffa6bac382398612b1"}
Jul 16 20:33:50.823970 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.823914    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4" event={"ID":"7bf1755d-416a-4bc5-92ce-ceac61fde3c5","Type":"ContainerDied","Data":"40ea2295f0648e1934abf6c3151c27f310ac2a5509884c65bf356f8ca306f944"}
Jul 16 20:33:50.823970 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.823938    2576 scope.go:117] "RemoveContainer" containerID="fcd11d7297dac050fa2cbdfa35bda8bd202590a55a3825ffa6bac382398612b1"
Jul 16 20:33:50.832981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.832965    2576 scope.go:117] "RemoveContainer" containerID="749631f90cc889aa89d5ba73031f6915eaa99e2d0f3133258e8bde3931b656b6"
Jul 16 20:33:50.840618 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.840602    2576 scope.go:117] "RemoveContainer" containerID="fcd11d7297dac050fa2cbdfa35bda8bd202590a55a3825ffa6bac382398612b1"
Jul 16 20:33:50.840930 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:33:50.840912    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd11d7297dac050fa2cbdfa35bda8bd202590a55a3825ffa6bac382398612b1\": container with ID starting with fcd11d7297dac050fa2cbdfa35bda8bd202590a55a3825ffa6bac382398612b1 not found: ID does not exist" containerID="fcd11d7297dac050fa2cbdfa35bda8bd202590a55a3825ffa6bac382398612b1"
Jul 16 20:33:50.841005 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.840941    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd11d7297dac050fa2cbdfa35bda8bd202590a55a3825ffa6bac382398612b1"} err="failed to get container status \"fcd11d7297dac050fa2cbdfa35bda8bd202590a55a3825ffa6bac382398612b1\": rpc error: code = NotFound desc = could not find container \"fcd11d7297dac050fa2cbdfa35bda8bd202590a55a3825ffa6bac382398612b1\": container with ID starting with fcd11d7297dac050fa2cbdfa35bda8bd202590a55a3825ffa6bac382398612b1 not found: ID does not exist"
Jul 16 20:33:50.841005 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.840963    2576 scope.go:117] "RemoveContainer" containerID="749631f90cc889aa89d5ba73031f6915eaa99e2d0f3133258e8bde3931b656b6"
Jul 16 20:33:50.841167 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:33:50.841151    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749631f90cc889aa89d5ba73031f6915eaa99e2d0f3133258e8bde3931b656b6\": container with ID starting with 749631f90cc889aa89d5ba73031f6915eaa99e2d0f3133258e8bde3931b656b6 not found: ID does not exist" containerID="749631f90cc889aa89d5ba73031f6915eaa99e2d0f3133258e8bde3931b656b6"
Jul 16 20:33:50.841222 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.841172    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749631f90cc889aa89d5ba73031f6915eaa99e2d0f3133258e8bde3931b656b6"} err="failed to get container status \"749631f90cc889aa89d5ba73031f6915eaa99e2d0f3133258e8bde3931b656b6\": rpc error: code = NotFound desc = could not find container \"749631f90cc889aa89d5ba73031f6915eaa99e2d0f3133258e8bde3931b656b6\": container with ID starting with 749631f90cc889aa89d5ba73031f6915eaa99e2d0f3133258e8bde3931b656b6 not found: ID does not exist"
Jul 16 20:33:50.846548 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.846523    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"]
Jul 16 20:33:50.850619 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.850597    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0d61d6f5-openshift-default-6b4cfd4cd7-sjrg4"]
Jul 16 20:33:50.852245 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.852228    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-envoy\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:33:50.852299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.852249    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-credential-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:33:50.852299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.852259    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-token\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:33:50.852299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.852269    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-podinfo\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:33:50.852299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.852277    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-workload-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:33:50.852299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.852285    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-istio-data\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:33:50.852299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.852293    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-workload-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:33:50.852299 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:50.852300    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k4tj\" (UniqueName: \"kubernetes.io/projected/7bf1755d-416a-4bc5-92ce-ceac61fde3c5-kube-api-access-8k4tj\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:33:52.245221 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:33:52.245184    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" path="/var/lib/kubelet/pods/7bf1755d-416a-4bc5-92ce-ceac61fde3c5/volumes"
Jul 16 20:35:20.822357 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.822320    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"]
Jul 16 20:35:20.822826 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.822667    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerName="istio-proxy"
Jul 16 20:35:20.822826 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.822684    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerName="istio-proxy"
Jul 16 20:35:20.822826 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.822725    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerName="istio-proxy"
Jul 16 20:35:20.822826 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.822735    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerName="istio-proxy"
Jul 16 20:35:20.822826 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.822797    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerName="istio-proxy"
Jul 16 20:35:20.822826 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.822806    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerName="istio-proxy"
Jul 16 20:35:20.822826 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.822815    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerName="istio-proxy"
Jul 16 20:35:20.822826 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.822821    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerName="istio-proxy"
Jul 16 20:35:20.823144 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.822925    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerName="istio-proxy"
Jul 16 20:35:20.823144 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.822935    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerName="istio-proxy"
Jul 16 20:35:20.823144 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.822942    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf1755d-416a-4bc5-92ce-ceac61fde3c5" containerName="istio-proxy"
Jul 16 20:35:20.826136 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.826118    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.828880 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.828859    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-087d0429-openshift-default-dockercfg-l2t4k\""
Jul 16 20:35:20.829414 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.829392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-podinfo\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.829470 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.829432    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-token\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.829549 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.829527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-credential-socket\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.829632 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.829612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-workload-socket\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.829723 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.829653    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istiod-ca-cert\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.829723 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.829690    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-workload-certs\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.829839 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.829732    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvfvd\" (UniqueName: \"kubernetes.io/projected/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-kube-api-access-dvfvd\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.829839 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.829770    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-data\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.829839 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.829805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-envoy\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.845413 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.845385    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"]
Jul 16 20:35:20.930230 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.930195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-podinfo\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.930230 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.930232    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-token\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.930501 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.930266    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-credential-socket\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.930501 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.930291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-workload-socket\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.930501 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.930306    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istiod-ca-cert\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.930501 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.930327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-workload-certs\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.930501 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.930351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvfvd\" (UniqueName: \"kubernetes.io/projected/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-kube-api-access-dvfvd\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.930501 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.930376    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-data\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.930501 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.930408    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-envoy\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.930946 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.930718    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-credential-socket\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.930946 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.930776    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-workload-socket\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.930946 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.930883    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-workload-certs\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.931090 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.931007    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-data\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.931242 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.931220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istiod-ca-cert\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.933031 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.933012    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-envoy\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.933318 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.933301    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-podinfo\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.938427 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.938400    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-token\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:20.938995 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:20.938977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvfvd\" (UniqueName: \"kubernetes.io/projected/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-kube-api-access-dvfvd\") pod \"e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") " pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:21.138093 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:21.137993    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:21.280103 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:21.280078    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"]
Jul 16 20:35:21.282064 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:35:21.282032    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74623f1_0986_4ae2_9ee8_49ea8b4ab9d0.slice/crio-ec48def007af94c227604e70cfe698527f2fc2ca0504d25bc22f924f86715067 WatchSource:0}: Error finding container ec48def007af94c227604e70cfe698527f2fc2ca0504d25bc22f924f86715067: Status 404 returned error can't find the container with id ec48def007af94c227604e70cfe698527f2fc2ca0504d25bc22f924f86715067
Jul 16 20:35:21.284062 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:21.284034    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:35:21.284130 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:21.284093    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:35:21.284130 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:21.284121    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:35:22.212164 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:22.212128    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz" event={"ID":"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0","Type":"ContainerStarted","Data":"9d3f7d66445a7c1b7054d5da5b88fca0c96ff8c856b99b39b06ed14ab1e2a97d"}
Jul 16 20:35:22.212523 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:22.212179    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz" event={"ID":"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0","Type":"ContainerStarted","Data":"ec48def007af94c227604e70cfe698527f2fc2ca0504d25bc22f924f86715067"}
Jul 16 20:35:22.233649 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:22.233576    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz" podStartSLOduration=2.233560796 podStartE2EDuration="2.233560796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:35:22.233024989 +0000 UTC m=+2100.483645749" watchObservedRunningTime="2026-07-16 20:35:22.233560796 +0000 UTC m=+2100.484181555"
Jul 16 20:35:23.138508 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:23.138482    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:23.143270 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:23.143246    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:23.216766 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:23.216734    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:23.217579 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:23.217563    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:38.218177 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:38.218094    2576 patch_prober.go:28] interesting pod/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 16 20:35:38.218177 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:38.218146    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz" podUID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 16 20:35:38.294224 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:38.294194    2576 generic.go:356] "Generic (PLEG): container finished" podID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" containerID="9d3f7d66445a7c1b7054d5da5b88fca0c96ff8c856b99b39b06ed14ab1e2a97d" exitCode=0
Jul 16 20:35:38.294397 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:38.294263    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz" event={"ID":"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0","Type":"ContainerDied","Data":"9d3f7d66445a7c1b7054d5da5b88fca0c96ff8c856b99b39b06ed14ab1e2a97d"}
Jul 16 20:35:38.294658 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:38.294644    2576 scope.go:117] "RemoveContainer" containerID="9d3f7d66445a7c1b7054d5da5b88fca0c96ff8c856b99b39b06ed14ab1e2a97d"
Jul 16 20:35:38.295669 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:38.295643    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:35:38.295780 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:38.295745    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:35:38.295834 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:38.295792    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:35:39.300654 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:39.300619    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz" event={"ID":"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0","Type":"ContainerStarted","Data":"7e1dd0be6694b5d9a5b29cdd84aff1239baefa049a029c672fedf560dbe2b7e7"}
Jul 16 20:35:40.138957 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:40.138926    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:40.139137 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:40.139120    2576 patch_prober.go:28] interesting pod/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 16 20:35:40.139182 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:40.139162    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz" podUID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 16 20:35:41.139231 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:41.139197    2576 patch_prober.go:28] interesting pod/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 16 20:35:41.139618 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:41.139276    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz" podUID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 16 20:35:42.138690 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:42.138653    2576 patch_prober.go:28] interesting pod/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 16 20:35:42.138887 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:42.138738    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz" podUID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 16 20:35:43.142766 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:43.142733    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:43.143212 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:43.143010    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:35:43.143725 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:35:43.143694    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:36:01.476826 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:01.476787    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"]
Jul 16 20:36:01.477320 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:01.477043    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz" podUID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" containerName="istio-proxy" containerID="cri-o://7e1dd0be6694b5d9a5b29cdd84aff1239baefa049a029c672fedf560dbe2b7e7" gracePeriod=30
Jul 16 20:36:06.727476 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.727447    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:36:06.738023 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738003    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-data\") pod \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") "
Jul 16 20:36:06.738108 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738063    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-credential-socket\") pod \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") "
Jul 16 20:36:06.738108 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738082    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-envoy\") pod \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") "
Jul 16 20:36:06.738180 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738151    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-podinfo\") pod \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") "
Jul 16 20:36:06.738221 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738179    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-token\") pod \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") "
Jul 16 20:36:06.738221 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738214    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvfvd\" (UniqueName: \"kubernetes.io/projected/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-kube-api-access-dvfvd\") pod \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") "
Jul 16 20:36:06.738350 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738325    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" (UID: "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:36:06.738415 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738368    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-workload-socket\") pod \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") "
Jul 16 20:36:06.738415 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738396    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istiod-ca-cert\") pod \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") "
Jul 16 20:36:06.738512 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738444    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-workload-certs\") pod \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\" (UID: \"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0\") "
Jul 16 20:36:06.738755 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738734    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-credential-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:36:06.738852 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738821    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" (UID: "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:36:06.739085 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738922    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" (UID: "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:36:06.739085 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.738939    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" (UID: "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:36:06.739317 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.739280    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" (UID: "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:36:06.740500 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.740465    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" (UID: "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:36:06.740608 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.740513    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" (UID: "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:36:06.740895 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.740876    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" (UID: "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 20:36:06.740952 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.740889    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-kube-api-access-dvfvd" (OuterVolumeSpecName: "kube-api-access-dvfvd") pod "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" (UID: "f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0"). InnerVolumeSpecName "kube-api-access-dvfvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:36:06.839129 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.839098    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-podinfo\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:36:06.839129 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.839122    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-token\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:36:06.839129 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.839131    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvfvd\" (UniqueName: \"kubernetes.io/projected/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-kube-api-access-dvfvd\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:36:06.839356 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.839141    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-workload-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:36:06.839356 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.839150    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istiod-ca-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:36:06.839356 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.839158    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-workload-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:36:06.839356 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.839166    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-data\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:36:06.839356 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:06.839173    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0-istio-envoy\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:36:07.420086 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:07.420050    2576 generic.go:356] "Generic (PLEG): container finished" podID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" containerID="7e1dd0be6694b5d9a5b29cdd84aff1239baefa049a029c672fedf560dbe2b7e7" exitCode=0
Jul 16 20:36:07.420277 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:07.420100    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"
Jul 16 20:36:07.420277 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:07.420107    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz" event={"ID":"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0","Type":"ContainerDied","Data":"7e1dd0be6694b5d9a5b29cdd84aff1239baefa049a029c672fedf560dbe2b7e7"}
Jul 16 20:36:07.420277 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:07.420159    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz" event={"ID":"f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0","Type":"ContainerDied","Data":"ec48def007af94c227604e70cfe698527f2fc2ca0504d25bc22f924f86715067"}
Jul 16 20:36:07.420277 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:07.420183    2576 scope.go:117] "RemoveContainer" containerID="7e1dd0be6694b5d9a5b29cdd84aff1239baefa049a029c672fedf560dbe2b7e7"
Jul 16 20:36:07.429535 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:07.429519    2576 scope.go:117] "RemoveContainer" containerID="9d3f7d66445a7c1b7054d5da5b88fca0c96ff8c856b99b39b06ed14ab1e2a97d"
Jul 16 20:36:07.437038 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:07.437023    2576 scope.go:117] "RemoveContainer" containerID="7e1dd0be6694b5d9a5b29cdd84aff1239baefa049a029c672fedf560dbe2b7e7"
Jul 16 20:36:07.437284 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:36:07.437264    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1dd0be6694b5d9a5b29cdd84aff1239baefa049a029c672fedf560dbe2b7e7\": container with ID starting with 7e1dd0be6694b5d9a5b29cdd84aff1239baefa049a029c672fedf560dbe2b7e7 not found: ID does not exist" containerID="7e1dd0be6694b5d9a5b29cdd84aff1239baefa049a029c672fedf560dbe2b7e7"
Jul 16 20:36:07.437340 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:07.437291    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1dd0be6694b5d9a5b29cdd84aff1239baefa049a029c672fedf560dbe2b7e7"} err="failed to get container status \"7e1dd0be6694b5d9a5b29cdd84aff1239baefa049a029c672fedf560dbe2b7e7\": rpc error: code = NotFound desc = could not find container \"7e1dd0be6694b5d9a5b29cdd84aff1239baefa049a029c672fedf560dbe2b7e7\": container with ID starting with 7e1dd0be6694b5d9a5b29cdd84aff1239baefa049a029c672fedf560dbe2b7e7 not found: ID does not exist"
Jul 16 20:36:07.437340 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:07.437308    2576 scope.go:117] "RemoveContainer" containerID="9d3f7d66445a7c1b7054d5da5b88fca0c96ff8c856b99b39b06ed14ab1e2a97d"
Jul 16 20:36:07.437542 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:36:07.437526    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3f7d66445a7c1b7054d5da5b88fca0c96ff8c856b99b39b06ed14ab1e2a97d\": container with ID starting with 9d3f7d66445a7c1b7054d5da5b88fca0c96ff8c856b99b39b06ed14ab1e2a97d not found: ID does not exist" containerID="9d3f7d66445a7c1b7054d5da5b88fca0c96ff8c856b99b39b06ed14ab1e2a97d"
Jul 16 20:36:07.437579 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:07.437548    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3f7d66445a7c1b7054d5da5b88fca0c96ff8c856b99b39b06ed14ab1e2a97d"} err="failed to get container status \"9d3f7d66445a7c1b7054d5da5b88fca0c96ff8c856b99b39b06ed14ab1e2a97d\": rpc error: code = NotFound desc = could not find container \"9d3f7d66445a7c1b7054d5da5b88fca0c96ff8c856b99b39b06ed14ab1e2a97d\": container with ID starting with 9d3f7d66445a7c1b7054d5da5b88fca0c96ff8c856b99b39b06ed14ab1e2a97d not found: ID does not exist"
Jul 16 20:36:07.441797 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:07.441771    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"]
Jul 16 20:36:07.446247 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:07.446227    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-087d0429-openshift-default-5486fd65b7-ltdvz"]
Jul 16 20:36:08.245421 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:08.245391    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" path="/var/lib/kubelet/pods/f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0/volumes"
Jul 16 20:36:21.606276 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.606194    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"]
Jul 16 20:36:21.606748 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.606536    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" containerName="istio-proxy"
Jul 16 20:36:21.606748 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.606547    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" containerName="istio-proxy"
Jul 16 20:36:21.606748 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.606565    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" containerName="istio-proxy"
Jul 16 20:36:21.606748 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.606572    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" containerName="istio-proxy"
Jul 16 20:36:21.606748 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.606632    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" containerName="istio-proxy"
Jul 16 20:36:21.606748 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.606641    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cf8253b-e921-45e1-b20c-cdcd38127ec9" containerName="istio-proxy"
Jul 16 20:36:21.610709 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.610679    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.613118 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.613095    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0faadf3f-openshift-default-dockercfg-xdmbd\""
Jul 16 20:36:21.620511 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.620490    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"]
Jul 16 20:36:21.652224 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.652160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-istio-envoy\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.652412 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.652393    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1713ae-d396-4351-8a62-b85729418567-istio-podinfo\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.652489 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.652446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-workload-certs\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.652489 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.652477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1713ae-d396-4351-8a62-b85729418567-istio-token\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.652629 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.652614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1713ae-d396-4351-8a62-b85729418567-istiod-ca-cert\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.652760 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.652746    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-workload-socket\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.652857 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.652786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-istio-data\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.652857 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.652831    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-credential-socket\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.652937 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.652855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2mj2\" (UniqueName: \"kubernetes.io/projected/5d1713ae-d396-4351-8a62-b85729418567-kube-api-access-s2mj2\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.753425 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.753392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-workload-socket\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.753425 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.753427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-istio-data\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.753714 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.753447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-credential-socket\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.753714 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.753465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2mj2\" (UniqueName: \"kubernetes.io/projected/5d1713ae-d396-4351-8a62-b85729418567-kube-api-access-s2mj2\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.753714 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.753491    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-istio-envoy\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.753714 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.753511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1713ae-d396-4351-8a62-b85729418567-istio-podinfo\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.753714 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.753546    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-workload-certs\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.753714 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.753573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1713ae-d396-4351-8a62-b85729418567-istio-token\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.753714 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.753608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1713ae-d396-4351-8a62-b85729418567-istiod-ca-cert\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.754033 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.753907    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-workload-socket\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.754033 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.753933    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-istio-data\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.754033 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.753962    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-credential-socket\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.754292 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.754274    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-workload-certs\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.754561 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.754543    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1713ae-d396-4351-8a62-b85729418567-istiod-ca-cert\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.756094 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.756060    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-istio-envoy\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.756594 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.756573    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1713ae-d396-4351-8a62-b85729418567-istio-podinfo\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.762600 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.762574    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1713ae-d396-4351-8a62-b85729418567-istio-token\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.762751 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.762628    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2mj2\" (UniqueName: \"kubernetes.io/projected/5d1713ae-d396-4351-8a62-b85729418567-kube-api-access-s2mj2\") pod \"e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") " pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:21.921453 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:21.921377    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:22.068037 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:22.068012    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"]
Jul 16 20:36:22.070523 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:36:22.070495    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1713ae_d396_4351_8a62_b85729418567.slice/crio-d792e4c7674df17cb99b938f415cf1c770989f9f3f00c5078423dec1f65dfdeb WatchSource:0}: Error finding container d792e4c7674df17cb99b938f415cf1c770989f9f3f00c5078423dec1f65dfdeb: Status 404 returned error can't find the container with id d792e4c7674df17cb99b938f415cf1c770989f9f3f00c5078423dec1f65dfdeb
Jul 16 20:36:22.072683 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:22.072653    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:36:22.072802 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:22.072757    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:36:22.072866 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:22.072804    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:36:22.486493 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:22.486448    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw" event={"ID":"5d1713ae-d396-4351-8a62-b85729418567","Type":"ContainerStarted","Data":"2db34c7e6cf9aabce2cda6faf6d0f6be977f1211b619f0ed744090b7f6d71c62"}
Jul 16 20:36:22.486672 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:22.486509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw" event={"ID":"5d1713ae-d396-4351-8a62-b85729418567","Type":"ContainerStarted","Data":"d792e4c7674df17cb99b938f415cf1c770989f9f3f00c5078423dec1f65dfdeb"}
Jul 16 20:36:22.513265 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:22.513212    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw" podStartSLOduration=1.5131960580000001 podStartE2EDuration="1.513196058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:36:22.511447196 +0000 UTC m=+2160.762067954" watchObservedRunningTime="2026-07-16 20:36:22.513196058 +0000 UTC m=+2160.763816816"
Jul 16 20:36:22.921689 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:22.921650    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:23.926763 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:23.926735    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:24.496008 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:24.495970    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:24.497033 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:24.497013    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:34.316281 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.316252    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"]
Jul 16 20:36:34.316923 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.316896    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f74623f1-0986-4ae2-9ee8-49ea8b4ab9d0" containerName="istio-proxy"
Jul 16 20:36:34.320213 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.320194    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.322844 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.322818    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0faadf3f\"/\"e2e-lifecycle-model-0faadf3f-kserve-self-signed-certs\""
Jul 16 20:36:34.323855 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.323828    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0faadf3f\"/\"kube-root-ca.crt\""
Jul 16 20:36:34.323964 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.323861    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0faadf3f\"/\"default-dockercfg-qdwdn\""
Jul 16 20:36:34.323964 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.323907    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0faadf3f\"/\"openshift-service-ca.crt\""
Jul 16 20:36:34.330083 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.330017    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"]
Jul 16 20:36:34.365017 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.364981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-dshm\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.365214 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.365034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-tmp-dir\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.365214 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.365056    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-home\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.365214 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.365163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4565406-1419-46f8-9d6e-8d27fbfe5089-tls-certs\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.365214 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.365197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs9lc\" (UniqueName: \"kubernetes.io/projected/f4565406-1419-46f8-9d6e-8d27fbfe5089-kube-api-access-qs9lc\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.365428 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.365286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-model-cache\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.466689 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.466646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4565406-1419-46f8-9d6e-8d27fbfe5089-tls-certs\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.466689 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.466683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs9lc\" (UniqueName: \"kubernetes.io/projected/f4565406-1419-46f8-9d6e-8d27fbfe5089-kube-api-access-qs9lc\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.466972 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.466754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-model-cache\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.466972 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.466806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-dshm\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.466972 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.466845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-tmp-dir\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.466972 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.466863    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-home\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.467383 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.467332    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-tmp-dir\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.467383 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.467372    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-model-cache\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.467578 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.467405    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-home\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.469873 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.469844    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-dshm\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.470006 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.469987    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4565406-1419-46f8-9d6e-8d27fbfe5089-tls-certs\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.474978 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.474958    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs9lc\" (UniqueName: \"kubernetes.io/projected/f4565406-1419-46f8-9d6e-8d27fbfe5089-kube-api-access-qs9lc\") pod \"e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") " pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.641358 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.641277    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:34.780361 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:34.780337    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"]
Jul 16 20:36:34.782803 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:36:34.782773    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4565406_1419_46f8_9d6e_8d27fbfe5089.slice/crio-804a9a00f7f9466d2ce80784ffaa61fdd5c79e6b86c48870199156b2c498c47b WatchSource:0}: Error finding container 804a9a00f7f9466d2ce80784ffaa61fdd5c79e6b86c48870199156b2c498c47b: Status 404 returned error can't find the container with id 804a9a00f7f9466d2ce80784ffaa61fdd5c79e6b86c48870199156b2c498c47b
Jul 16 20:36:35.548672 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:35.548632    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2" event={"ID":"f4565406-1419-46f8-9d6e-8d27fbfe5089","Type":"ContainerStarted","Data":"68bcb8f1b883c9e3df36043e7253c72af8032e69fe50f607de23d3f06dbfcb96"}
Jul 16 20:36:35.549076 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:35.548682    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2" event={"ID":"f4565406-1419-46f8-9d6e-8d27fbfe5089","Type":"ContainerStarted","Data":"804a9a00f7f9466d2ce80784ffaa61fdd5c79e6b86c48870199156b2c498c47b"}
Jul 16 20:36:35.566394 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:35.566344    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2" podStartSLOduration=1.5663291369999999 podStartE2EDuration="1.566329137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:36:35.564773418 +0000 UTC m=+2173.815394179" watchObservedRunningTime="2026-07-16 20:36:35.566329137 +0000 UTC m=+2173.816949896"
Jul 16 20:36:36.554156 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:36.554116    2576 generic.go:356] "Generic (PLEG): container finished" podID="5d1713ae-d396-4351-8a62-b85729418567" containerID="2db34c7e6cf9aabce2cda6faf6d0f6be977f1211b619f0ed744090b7f6d71c62" exitCode=0
Jul 16 20:36:36.554673 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:36.554192    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw" event={"ID":"5d1713ae-d396-4351-8a62-b85729418567","Type":"ContainerDied","Data":"2db34c7e6cf9aabce2cda6faf6d0f6be977f1211b619f0ed744090b7f6d71c62"}
Jul 16 20:36:36.554774 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:36.554686    2576 scope.go:117] "RemoveContainer" containerID="2db34c7e6cf9aabce2cda6faf6d0f6be977f1211b619f0ed744090b7f6d71c62"
Jul 16 20:36:36.555694 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:36.555661    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:36:36.555833 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:36.555767    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:36:36.555894 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:36.555831    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:36:37.563035 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:37.562987    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw" event={"ID":"5d1713ae-d396-4351-8a62-b85729418567","Type":"ContainerStarted","Data":"12c674de82c7d32835d0ac76718073c0ca09966b7891592c3345798930f13e9d"}
Jul 16 20:36:37.922463 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:37.922364    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:37.923557 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:37.923522    2576 patch_prober.go:28] interesting pod/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 16 20:36:37.923669 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:37.923581    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw" podUID="5d1713ae-d396-4351-8a62-b85729418567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 16 20:36:38.921905 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:38.921868    2576 patch_prober.go:28] interesting pod/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 16 20:36:38.922296 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:38.921926    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw" podUID="5d1713ae-d396-4351-8a62-b85729418567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 16 20:36:39.496867 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:39.496826    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:39.922051 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:39.922014    2576 patch_prober.go:28] interesting pod/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 16 20:36:39.922441 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:39.922079    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw" podUID="5d1713ae-d396-4351-8a62-b85729418567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 16 20:36:40.922203 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:40.922171    2576 patch_prober.go:28] interesting pod/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 16 20:36:40.922588 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:40.922222    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw" podUID="5d1713ae-d396-4351-8a62-b85729418567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 16 20:36:41.926192 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:41.926163    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:41.927211 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:41.927195    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:36:44.642153 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:44.642116    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:44.642536 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:44.642173    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:44.654984 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:44.654963    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:36:45.616011 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:36:45.615984    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:37:21.872011 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:21.871975    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"]
Jul 16 20:37:36.892405 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:36.892346    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2" podUID="f4565406-1419-46f8-9d6e-8d27fbfe5089" containerName="main" containerID="cri-o://68bcb8f1b883c9e3df36043e7253c72af8032e69fe50f607de23d3f06dbfcb96" gracePeriod=45
Jul 16 20:37:37.133254 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.133232    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:37:37.233695 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.233617    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-home\") pod \"f4565406-1419-46f8-9d6e-8d27fbfe5089\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") "
Jul 16 20:37:37.233695 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.233662    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-dshm\") pod \"f4565406-1419-46f8-9d6e-8d27fbfe5089\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") "
Jul 16 20:37:37.233928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.233756    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-tmp-dir\") pod \"f4565406-1419-46f8-9d6e-8d27fbfe5089\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") "
Jul 16 20:37:37.233928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.233798    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs9lc\" (UniqueName: \"kubernetes.io/projected/f4565406-1419-46f8-9d6e-8d27fbfe5089-kube-api-access-qs9lc\") pod \"f4565406-1419-46f8-9d6e-8d27fbfe5089\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") "
Jul 16 20:37:37.233928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.233847    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4565406-1419-46f8-9d6e-8d27fbfe5089-tls-certs\") pod \"f4565406-1419-46f8-9d6e-8d27fbfe5089\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") "
Jul 16 20:37:37.233928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.233881    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-model-cache\") pod \"f4565406-1419-46f8-9d6e-8d27fbfe5089\" (UID: \"f4565406-1419-46f8-9d6e-8d27fbfe5089\") "
Jul 16 20:37:37.233928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.233935    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-home" (OuterVolumeSpecName: "home") pod "f4565406-1419-46f8-9d6e-8d27fbfe5089" (UID: "f4565406-1419-46f8-9d6e-8d27fbfe5089"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:37:37.234234 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.234038    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4565406-1419-46f8-9d6e-8d27fbfe5089" (UID: "f4565406-1419-46f8-9d6e-8d27fbfe5089"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:37:37.234234 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.234203    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4565406-1419-46f8-9d6e-8d27fbfe5089" (UID: "f4565406-1419-46f8-9d6e-8d27fbfe5089"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:37:37.234322 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.234245    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-tmp-dir\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:37:37.234322 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.234265    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-home\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:37:37.236074 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.236051    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-dshm" (OuterVolumeSpecName: "dshm") pod "f4565406-1419-46f8-9d6e-8d27fbfe5089" (UID: "f4565406-1419-46f8-9d6e-8d27fbfe5089"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:37:37.236074 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.236059    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4565406-1419-46f8-9d6e-8d27fbfe5089-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4565406-1419-46f8-9d6e-8d27fbfe5089" (UID: "f4565406-1419-46f8-9d6e-8d27fbfe5089"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:37:37.236222 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.236153    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4565406-1419-46f8-9d6e-8d27fbfe5089-kube-api-access-qs9lc" (OuterVolumeSpecName: "kube-api-access-qs9lc") pod "f4565406-1419-46f8-9d6e-8d27fbfe5089" (UID: "f4565406-1419-46f8-9d6e-8d27fbfe5089"). InnerVolumeSpecName "kube-api-access-qs9lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:37:37.335655 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.335616    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs9lc\" (UniqueName: \"kubernetes.io/projected/f4565406-1419-46f8-9d6e-8d27fbfe5089-kube-api-access-qs9lc\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:37:37.335655 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.335653    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4565406-1419-46f8-9d6e-8d27fbfe5089-tls-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:37:37.335943 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.335670    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-model-cache\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:37:37.335943 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.335682    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4565406-1419-46f8-9d6e-8d27fbfe5089-dshm\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:37:37.832685 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.832649    2576 generic.go:356] "Generic (PLEG): container finished" podID="f4565406-1419-46f8-9d6e-8d27fbfe5089" containerID="68bcb8f1b883c9e3df36043e7253c72af8032e69fe50f607de23d3f06dbfcb96" exitCode=0
Jul 16 20:37:37.832868 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.832721    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"
Jul 16 20:37:37.832868 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.832747    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2" event={"ID":"f4565406-1419-46f8-9d6e-8d27fbfe5089","Type":"ContainerDied","Data":"68bcb8f1b883c9e3df36043e7253c72af8032e69fe50f607de23d3f06dbfcb96"}
Jul 16 20:37:37.832868 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.832793    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2" event={"ID":"f4565406-1419-46f8-9d6e-8d27fbfe5089","Type":"ContainerDied","Data":"804a9a00f7f9466d2ce80784ffaa61fdd5c79e6b86c48870199156b2c498c47b"}
Jul 16 20:37:37.832868 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.832810    2576 scope.go:117] "RemoveContainer" containerID="68bcb8f1b883c9e3df36043e7253c72af8032e69fe50f607de23d3f06dbfcb96"
Jul 16 20:37:37.841753 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.841734    2576 scope.go:117] "RemoveContainer" containerID="68bcb8f1b883c9e3df36043e7253c72af8032e69fe50f607de23d3f06dbfcb96"
Jul 16 20:37:37.842024 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:37:37.842004    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68bcb8f1b883c9e3df36043e7253c72af8032e69fe50f607de23d3f06dbfcb96\": container with ID starting with 68bcb8f1b883c9e3df36043e7253c72af8032e69fe50f607de23d3f06dbfcb96 not found: ID does not exist" containerID="68bcb8f1b883c9e3df36043e7253c72af8032e69fe50f607de23d3f06dbfcb96"
Jul 16 20:37:37.842098 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.842032    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68bcb8f1b883c9e3df36043e7253c72af8032e69fe50f607de23d3f06dbfcb96"} err="failed to get container status \"68bcb8f1b883c9e3df36043e7253c72af8032e69fe50f607de23d3f06dbfcb96\": rpc error: code = NotFound desc = could not find container \"68bcb8f1b883c9e3df36043e7253c72af8032e69fe50f607de23d3f06dbfcb96\": container with ID starting with 68bcb8f1b883c9e3df36043e7253c72af8032e69fe50f607de23d3f06dbfcb96 not found: ID does not exist"
Jul 16 20:37:37.851986 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.851962    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"]
Jul 16 20:37:37.855597 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:37.855576    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-0faadf3f/e2e-lifecycle-model-0faadf3f-kserve-f77c6f979-wrbf2"]
Jul 16 20:37:38.245735 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:37:38.245653    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4565406-1419-46f8-9d6e-8d27fbfe5089" path="/var/lib/kubelet/pods/f4565406-1419-46f8-9d6e-8d27fbfe5089/volumes"
Jul 16 20:38:09.029572 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:09.029528    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"]
Jul 16 20:38:09.030148 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:09.029796    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw" podUID="5d1713ae-d396-4351-8a62-b85729418567" containerName="istio-proxy" containerID="cri-o://12c674de82c7d32835d0ac76718073c0ca09966b7891592c3345798930f13e9d" gracePeriod=30
Jul 16 20:38:11.928838 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:11.928801    2576 patch_prober.go:28] interesting pod/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 20:38:11.929306 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:11.928877    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw" podUID="5d1713ae-d396-4351-8a62-b85729418567" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 20:38:14.276405 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.276377    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:38:14.348470 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.348444    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1713ae-d396-4351-8a62-b85729418567-istio-token\") pod \"5d1713ae-d396-4351-8a62-b85729418567\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") "
Jul 16 20:38:14.348662 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.348496    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-istio-data\") pod \"5d1713ae-d396-4351-8a62-b85729418567\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") "
Jul 16 20:38:14.348662 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.348520    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1713ae-d396-4351-8a62-b85729418567-istio-podinfo\") pod \"5d1713ae-d396-4351-8a62-b85729418567\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") "
Jul 16 20:38:14.348662 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.348539    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-workload-socket\") pod \"5d1713ae-d396-4351-8a62-b85729418567\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") "
Jul 16 20:38:14.348662 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.348598    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-workload-certs\") pod \"5d1713ae-d396-4351-8a62-b85729418567\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") "
Jul 16 20:38:14.348662 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.348614    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1713ae-d396-4351-8a62-b85729418567-istiod-ca-cert\") pod \"5d1713ae-d396-4351-8a62-b85729418567\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") "
Jul 16 20:38:14.348662 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.348638    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-credential-socket\") pod \"5d1713ae-d396-4351-8a62-b85729418567\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") "
Jul 16 20:38:14.349024 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.348668    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-istio-envoy\") pod \"5d1713ae-d396-4351-8a62-b85729418567\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") "
Jul 16 20:38:14.349024 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.348977    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d1713ae-d396-4351-8a62-b85729418567" (UID: "5d1713ae-d396-4351-8a62-b85729418567"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:38:14.349024 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.348998    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d1713ae-d396-4351-8a62-b85729418567" (UID: "5d1713ae-d396-4351-8a62-b85729418567"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:38:14.349281 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.349067    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2mj2\" (UniqueName: \"kubernetes.io/projected/5d1713ae-d396-4351-8a62-b85729418567-kube-api-access-s2mj2\") pod \"5d1713ae-d396-4351-8a62-b85729418567\" (UID: \"5d1713ae-d396-4351-8a62-b85729418567\") "
Jul 16 20:38:14.349281 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.349088    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d1713ae-d396-4351-8a62-b85729418567" (UID: "5d1713ae-d396-4351-8a62-b85729418567"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:38:14.349461 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.349300    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1713ae-d396-4351-8a62-b85729418567-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d1713ae-d396-4351-8a62-b85729418567" (UID: "5d1713ae-d396-4351-8a62-b85729418567"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:38:14.349461 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.349404    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d1713ae-d396-4351-8a62-b85729418567" (UID: "5d1713ae-d396-4351-8a62-b85729418567"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:38:14.349574 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.349530    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-istio-data\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:38:14.349574 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.349552    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-workload-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:38:14.349574 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.349568    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-workload-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:38:14.349756 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.349582    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1713ae-d396-4351-8a62-b85729418567-istiod-ca-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:38:14.349756 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.349596    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-credential-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:38:14.350853 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.350832    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1713ae-d396-4351-8a62-b85729418567-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d1713ae-d396-4351-8a62-b85729418567" (UID: "5d1713ae-d396-4351-8a62-b85729418567"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:38:14.350853 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.350844    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d1713ae-d396-4351-8a62-b85729418567-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d1713ae-d396-4351-8a62-b85729418567" (UID: "5d1713ae-d396-4351-8a62-b85729418567"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 20:38:14.351028 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.351013    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d1713ae-d396-4351-8a62-b85729418567" (UID: "5d1713ae-d396-4351-8a62-b85729418567"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:38:14.351301 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.351288    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1713ae-d396-4351-8a62-b85729418567-kube-api-access-s2mj2" (OuterVolumeSpecName: "kube-api-access-s2mj2") pod "5d1713ae-d396-4351-8a62-b85729418567" (UID: "5d1713ae-d396-4351-8a62-b85729418567"). InnerVolumeSpecName "kube-api-access-s2mj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:38:14.450055 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.450016    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1713ae-d396-4351-8a62-b85729418567-istio-podinfo\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:38:14.450055 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.450043    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1713ae-d396-4351-8a62-b85729418567-istio-envoy\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:38:14.450055 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.450061    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2mj2\" (UniqueName: \"kubernetes.io/projected/5d1713ae-d396-4351-8a62-b85729418567-kube-api-access-s2mj2\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:38:14.450291 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.450071    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1713ae-d396-4351-8a62-b85729418567-istio-token\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:38:14.988567 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.988534    2576 generic.go:356] "Generic (PLEG): container finished" podID="5d1713ae-d396-4351-8a62-b85729418567" containerID="12c674de82c7d32835d0ac76718073c0ca09966b7891592c3345798930f13e9d" exitCode=0
Jul 16 20:38:14.988797 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.988592    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"
Jul 16 20:38:14.988797 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.988627    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw" event={"ID":"5d1713ae-d396-4351-8a62-b85729418567","Type":"ContainerDied","Data":"12c674de82c7d32835d0ac76718073c0ca09966b7891592c3345798930f13e9d"}
Jul 16 20:38:14.988797 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.988692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw" event={"ID":"5d1713ae-d396-4351-8a62-b85729418567","Type":"ContainerDied","Data":"d792e4c7674df17cb99b938f415cf1c770989f9f3f00c5078423dec1f65dfdeb"}
Jul 16 20:38:14.988797 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.988729    2576 scope.go:117] "RemoveContainer" containerID="12c674de82c7d32835d0ac76718073c0ca09966b7891592c3345798930f13e9d"
Jul 16 20:38:14.997795 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:14.997774    2576 scope.go:117] "RemoveContainer" containerID="2db34c7e6cf9aabce2cda6faf6d0f6be977f1211b619f0ed744090b7f6d71c62"
Jul 16 20:38:15.006194 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:15.006176    2576 scope.go:117] "RemoveContainer" containerID="12c674de82c7d32835d0ac76718073c0ca09966b7891592c3345798930f13e9d"
Jul 16 20:38:15.006457 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:38:15.006438    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c674de82c7d32835d0ac76718073c0ca09966b7891592c3345798930f13e9d\": container with ID starting with 12c674de82c7d32835d0ac76718073c0ca09966b7891592c3345798930f13e9d not found: ID does not exist" containerID="12c674de82c7d32835d0ac76718073c0ca09966b7891592c3345798930f13e9d"
Jul 16 20:38:15.006509 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:15.006467    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c674de82c7d32835d0ac76718073c0ca09966b7891592c3345798930f13e9d"} err="failed to get container status \"12c674de82c7d32835d0ac76718073c0ca09966b7891592c3345798930f13e9d\": rpc error: code = NotFound desc = could not find container \"12c674de82c7d32835d0ac76718073c0ca09966b7891592c3345798930f13e9d\": container with ID starting with 12c674de82c7d32835d0ac76718073c0ca09966b7891592c3345798930f13e9d not found: ID does not exist"
Jul 16 20:38:15.006509 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:15.006486    2576 scope.go:117] "RemoveContainer" containerID="2db34c7e6cf9aabce2cda6faf6d0f6be977f1211b619f0ed744090b7f6d71c62"
Jul 16 20:38:15.006738 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:38:15.006722    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db34c7e6cf9aabce2cda6faf6d0f6be977f1211b619f0ed744090b7f6d71c62\": container with ID starting with 2db34c7e6cf9aabce2cda6faf6d0f6be977f1211b619f0ed744090b7f6d71c62 not found: ID does not exist" containerID="2db34c7e6cf9aabce2cda6faf6d0f6be977f1211b619f0ed744090b7f6d71c62"
Jul 16 20:38:15.006799 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:15.006751    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db34c7e6cf9aabce2cda6faf6d0f6be977f1211b619f0ed744090b7f6d71c62"} err="failed to get container status \"2db34c7e6cf9aabce2cda6faf6d0f6be977f1211b619f0ed744090b7f6d71c62\": rpc error: code = NotFound desc = could not find container \"2db34c7e6cf9aabce2cda6faf6d0f6be977f1211b619f0ed744090b7f6d71c62\": container with ID starting with 2db34c7e6cf9aabce2cda6faf6d0f6be977f1211b619f0ed744090b7f6d71c62 not found: ID does not exist"
Jul 16 20:38:15.014926 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:15.010851    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"]
Jul 16 20:38:15.015818 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:15.015770    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0faadf3f-openshift-default-56756d6c-fpfdw"]
Jul 16 20:38:16.243886 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:38:16.243848    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d1713ae-d396-4351-8a62-b85729418567" path="/var/lib/kubelet/pods/5d1713ae-d396-4351-8a62-b85729418567/volumes"
Jul 16 20:39:03.463146 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.463115    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"]
Jul 16 20:39:03.463619 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.463486    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d1713ae-d396-4351-8a62-b85729418567" containerName="istio-proxy"
Jul 16 20:39:03.463619 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.463498    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d1713ae-d396-4351-8a62-b85729418567" containerName="istio-proxy"
Jul 16 20:39:03.463619 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.463507    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4565406-1419-46f8-9d6e-8d27fbfe5089" containerName="main"
Jul 16 20:39:03.463619 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.463512    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4565406-1419-46f8-9d6e-8d27fbfe5089" containerName="main"
Jul 16 20:39:03.463619 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.463581    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d1713ae-d396-4351-8a62-b85729418567" containerName="istio-proxy"
Jul 16 20:39:03.463619 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.463589    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4565406-1419-46f8-9d6e-8d27fbfe5089" containerName="main"
Jul 16 20:39:03.466448 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.466429    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.469415 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.469390    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-955113-openshift-default-dockercfg-56j5z\""
Jul 16 20:39:03.485827 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.485789    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"]
Jul 16 20:39:03.576243 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.576210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-podinfo\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.576416 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.576274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-credential-socket\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.576416 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.576299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-envoy\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.576416 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.576314    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istiod-ca-cert\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.576416 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.576336    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szs4p\" (UniqueName: \"kubernetes.io/projected/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-kube-api-access-szs4p\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.576564 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.576444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-workload-certs\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.576564 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.576471    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-data\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.576564 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.576505    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-token\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.576660 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.576561    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-workload-socket\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.677312 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.677276    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-credential-socket\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.677497 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.677317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-envoy\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.677497 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.677342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istiod-ca-cert\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.677620 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.677515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szs4p\" (UniqueName: \"kubernetes.io/projected/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-kube-api-access-szs4p\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.677620 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.677586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-workload-certs\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.677753 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.677617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-data\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.677753 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.677667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-token\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.677753 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.677735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-workload-socket\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.677753 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.677745    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-credential-socket\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.677965 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.677827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-podinfo\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.678017 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.677972    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-workload-certs\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.678109 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.678076    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-workload-socket\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.678258 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.678237    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-data\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.678258 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.678246    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istiod-ca-cert\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.680271 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.680241    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-envoy\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.680468 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.680449    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-podinfo\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.686726 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.686684    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szs4p\" (UniqueName: \"kubernetes.io/projected/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-kube-api-access-szs4p\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.687047 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.687027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-token\") pod \"e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") " pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.783787 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.783681    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:03.925900 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.925865    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"]
Jul 16 20:39:03.928592 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:39:03.928566    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652e8a0f_cb48_42f4_b1ef_8285fe68fe83.slice/crio-c9275b3a268cf9bc8d7dd30be713388be55d74dbd63181987925feca2dde89fa WatchSource:0}: Error finding container c9275b3a268cf9bc8d7dd30be713388be55d74dbd63181987925feca2dde89fa: Status 404 returned error can't find the container with id c9275b3a268cf9bc8d7dd30be713388be55d74dbd63181987925feca2dde89fa
Jul 16 20:39:03.931249 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.931222    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:39:03.931323 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.931290    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:39:03.931365 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:03.931325    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:39:04.206599 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:04.206565    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" event={"ID":"652e8a0f-cb48-42f4-b1ef-8285fe68fe83","Type":"ContainerStarted","Data":"6c375cd0e1cd753e080d8b19eadcf5821fdcd61b04c702ffb3d8ab71b98c34e3"}
Jul 16 20:39:04.206795 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:04.206627    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" event={"ID":"652e8a0f-cb48-42f4-b1ef-8285fe68fe83","Type":"ContainerStarted","Data":"c9275b3a268cf9bc8d7dd30be713388be55d74dbd63181987925feca2dde89fa"}
Jul 16 20:39:04.227600 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:04.227551    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" podStartSLOduration=1.227536598 podStartE2EDuration="1.227536598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:04.226271636 +0000 UTC m=+2322.476892396" watchObservedRunningTime="2026-07-16 20:39:04.227536598 +0000 UTC m=+2322.478157356"
Jul 16 20:39:04.784530 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:04.784496    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:04.789592 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:04.789569    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:05.211087 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:05.211050    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:05.212116 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:05.212096    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:20.219595 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:20.219565    2576 patch_prober.go:28] interesting pod/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:51734->10.133.0.41:15021: read: connection reset by peer" start-of-body=
Jul 16 20:39:20.219963 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:20.219638    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:51734->10.133.0.41:15021: read: connection reset by peer"
Jul 16 20:39:20.283192 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:20.283164    2576 generic.go:356] "Generic (PLEG): container finished" podID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerID="6c375cd0e1cd753e080d8b19eadcf5821fdcd61b04c702ffb3d8ab71b98c34e3" exitCode=0
Jul 16 20:39:20.283338 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:20.283239    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" event={"ID":"652e8a0f-cb48-42f4-b1ef-8285fe68fe83","Type":"ContainerDied","Data":"6c375cd0e1cd753e080d8b19eadcf5821fdcd61b04c702ffb3d8ab71b98c34e3"}
Jul 16 20:39:20.283615 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:20.283602    2576 scope.go:117] "RemoveContainer" containerID="6c375cd0e1cd753e080d8b19eadcf5821fdcd61b04c702ffb3d8ab71b98c34e3"
Jul 16 20:39:20.284559 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:20.284535    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:39:20.284608 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:20.284599    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:39:20.284647 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:20.284633    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:39:21.289923 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:21.289885    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" event={"ID":"652e8a0f-cb48-42f4-b1ef-8285fe68fe83","Type":"ContainerStarted","Data":"a85f9ed52636506a7c0cedb10bd3ab4c725f95ab467e1913eec3ef0b7a170da6"}
Jul 16 20:39:21.784372 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:21.784340    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:21.784553 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:21.784528    2576 patch_prober.go:28] interesting pod/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 16 20:39:21.784599 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:21.784572    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 16 20:39:22.784301 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:22.784263    2576 patch_prober.go:28] interesting pod/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 16 20:39:22.784670 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:22.784337    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 16 20:39:23.784602 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:23.784560    2576 patch_prober.go:28] interesting pod/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 16 20:39:23.785014 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:23.784617    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 16 20:39:24.785052 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:24.785014    2576 patch_prober.go:28] interesting pod/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 16 20:39:24.785462 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:24.785069    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 16 20:39:25.785025 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:25.784985    2576 patch_prober.go:28] interesting pod/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 16 20:39:25.785204 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:25.785041    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 16 20:39:26.785189 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:26.785037    2576 patch_prober.go:28] interesting pod/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 16 20:39:26.785189 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:26.785096    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 16 20:39:27.788349 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:27.788318    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:27.788760 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:27.788612    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:27.789202 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:27.789183    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:39:41.022385 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.022346    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"]
Jul 16 20:39:41.022845 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.022678    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d1713ae-d396-4351-8a62-b85729418567" containerName="istio-proxy"
Jul 16 20:39:41.022845 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.022690    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d1713ae-d396-4351-8a62-b85729418567" containerName="istio-proxy"
Jul 16 20:39:41.022845 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.022782    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d1713ae-d396-4351-8a62-b85729418567" containerName="istio-proxy"
Jul 16 20:39:41.025562 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.025546    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.029056 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.029033    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-955113\"/\"kube-root-ca.crt\""
Jul 16 20:39:41.029056 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.029045    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-955113\"/\"test-model-955113-kserve-self-signed-certs\""
Jul 16 20:39:41.029056 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.029054    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-955113\"/\"openshift-service-ca.crt\""
Jul 16 20:39:41.029285 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.029055    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-955113\"/\"default-dockercfg-t5qkn\""
Jul 16 20:39:41.036440 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.036413    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"]
Jul 16 20:39:41.111828 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.111795    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-model-cache\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.111978 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.111857    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cclmr\" (UniqueName: \"kubernetes.io/projected/90b0c481-5330-4991-b18f-8a02bf87cd42-kube-api-access-cclmr\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.111978 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.111888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-tmp-dir\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.111978 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.111921    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-home\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.112088 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.111979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-dshm\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.112088 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.112040    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b0c481-5330-4991-b18f-8a02bf87cd42-tls-certs\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.212434 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.212403    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cclmr\" (UniqueName: \"kubernetes.io/projected/90b0c481-5330-4991-b18f-8a02bf87cd42-kube-api-access-cclmr\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.212599 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.212441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-tmp-dir\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.212599 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.212478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-home\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.212599 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.212499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-dshm\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.212599 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.212524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b0c481-5330-4991-b18f-8a02bf87cd42-tls-certs\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.212599 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.212553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-model-cache\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.212987 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.212963    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-home\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.213071 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.212985    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-model-cache\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.213071 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.213000    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-tmp-dir\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.214977 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.214960    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-dshm\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.215298 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.215281    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b0c481-5330-4991-b18f-8a02bf87cd42-tls-certs\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.221131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.221109    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cclmr\" (UniqueName: \"kubernetes.io/projected/90b0c481-5330-4991-b18f-8a02bf87cd42-kube-api-access-cclmr\") pod \"test-model-955113-kserve-847cff4d65-rjvcv\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") " pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.345896 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.345866    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:41.480020 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:41.479992    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"]
Jul 16 20:39:41.482596 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:39:41.482562    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90b0c481_5330_4991_b18f_8a02bf87cd42.slice/crio-800fc303d365379103a8e155cfe33e3ced36d0a3d6814b623663343e62744297 WatchSource:0}: Error finding container 800fc303d365379103a8e155cfe33e3ced36d0a3d6814b623663343e62744297: Status 404 returned error can't find the container with id 800fc303d365379103a8e155cfe33e3ced36d0a3d6814b623663343e62744297
Jul 16 20:39:42.385160 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:42.385127    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv" event={"ID":"90b0c481-5330-4991-b18f-8a02bf87cd42","Type":"ContainerStarted","Data":"d7fb9366d1d806ed7259d87f57586b936f45361899cab24556259582e70c1eaf"}
Jul 16 20:39:42.385160 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:42.385167    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv" event={"ID":"90b0c481-5330-4991-b18f-8a02bf87cd42","Type":"ContainerStarted","Data":"800fc303d365379103a8e155cfe33e3ced36d0a3d6814b623663343e62744297"}
Jul 16 20:39:42.403204 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:42.403155    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv" podStartSLOduration=1.403141065 podStartE2EDuration="1.403141065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:42.400776073 +0000 UTC m=+2360.651396831" watchObservedRunningTime="2026-07-16 20:39:42.403141065 +0000 UTC m=+2360.653761824"
Jul 16 20:39:51.346285 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:51.346247    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:51.346874 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:51.346310    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:51.358614 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:51.358587    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:39:51.438603 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:39:51.438573    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:40:33.168022 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:33.167987    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"]
Jul 16 20:40:48.188379 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.188289    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv" podUID="90b0c481-5330-4991-b18f-8a02bf87cd42" containerName="main" containerID="cri-o://d7fb9366d1d806ed7259d87f57586b936f45361899cab24556259582e70c1eaf" gracePeriod=45
Jul 16 20:40:48.444302 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.444235    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:40:48.517912 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.517875    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-tmp-dir\") pod \"90b0c481-5330-4991-b18f-8a02bf87cd42\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") "
Jul 16 20:40:48.518089 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.517923    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cclmr\" (UniqueName: \"kubernetes.io/projected/90b0c481-5330-4991-b18f-8a02bf87cd42-kube-api-access-cclmr\") pod \"90b0c481-5330-4991-b18f-8a02bf87cd42\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") "
Jul 16 20:40:48.518089 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.517949    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b0c481-5330-4991-b18f-8a02bf87cd42-tls-certs\") pod \"90b0c481-5330-4991-b18f-8a02bf87cd42\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") "
Jul 16 20:40:48.518089 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.517975    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-model-cache\") pod \"90b0c481-5330-4991-b18f-8a02bf87cd42\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") "
Jul 16 20:40:48.518089 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.518010    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-dshm\") pod \"90b0c481-5330-4991-b18f-8a02bf87cd42\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") "
Jul 16 20:40:48.518310 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.518098    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-home\") pod \"90b0c481-5330-4991-b18f-8a02bf87cd42\" (UID: \"90b0c481-5330-4991-b18f-8a02bf87cd42\") "
Jul 16 20:40:48.518310 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.518199    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90b0c481-5330-4991-b18f-8a02bf87cd42" (UID: "90b0c481-5330-4991-b18f-8a02bf87cd42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:40:48.518420 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.518352    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-model-cache" (OuterVolumeSpecName: "model-cache") pod "90b0c481-5330-4991-b18f-8a02bf87cd42" (UID: "90b0c481-5330-4991-b18f-8a02bf87cd42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:40:48.518420 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.518389    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-tmp-dir\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:40:48.518420 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.518385    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-home" (OuterVolumeSpecName: "home") pod "90b0c481-5330-4991-b18f-8a02bf87cd42" (UID: "90b0c481-5330-4991-b18f-8a02bf87cd42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:40:48.520332 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.520303    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-dshm" (OuterVolumeSpecName: "dshm") pod "90b0c481-5330-4991-b18f-8a02bf87cd42" (UID: "90b0c481-5330-4991-b18f-8a02bf87cd42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:40:48.520332 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.520317    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b0c481-5330-4991-b18f-8a02bf87cd42-kube-api-access-cclmr" (OuterVolumeSpecName: "kube-api-access-cclmr") pod "90b0c481-5330-4991-b18f-8a02bf87cd42" (UID: "90b0c481-5330-4991-b18f-8a02bf87cd42"). InnerVolumeSpecName "kube-api-access-cclmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:40:48.520477 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.520368    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b0c481-5330-4991-b18f-8a02bf87cd42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90b0c481-5330-4991-b18f-8a02bf87cd42" (UID: "90b0c481-5330-4991-b18f-8a02bf87cd42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:40:48.619048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.619009    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-dshm\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:40:48.619048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.619041    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-home\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:40:48.619048 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.619054    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cclmr\" (UniqueName: \"kubernetes.io/projected/90b0c481-5330-4991-b18f-8a02bf87cd42-kube-api-access-cclmr\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:40:48.619291 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.619068    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b0c481-5330-4991-b18f-8a02bf87cd42-tls-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:40:48.619291 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.619080    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b0c481-5330-4991-b18f-8a02bf87cd42-model-cache\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:40:48.671562 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.671532    2576 generic.go:356] "Generic (PLEG): container finished" podID="90b0c481-5330-4991-b18f-8a02bf87cd42" containerID="d7fb9366d1d806ed7259d87f57586b936f45361899cab24556259582e70c1eaf" exitCode=0
Jul 16 20:40:48.671761 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.671574    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"
Jul 16 20:40:48.671761 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.671579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv" event={"ID":"90b0c481-5330-4991-b18f-8a02bf87cd42","Type":"ContainerDied","Data":"d7fb9366d1d806ed7259d87f57586b936f45361899cab24556259582e70c1eaf"}
Jul 16 20:40:48.671761 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.671606    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv" event={"ID":"90b0c481-5330-4991-b18f-8a02bf87cd42","Type":"ContainerDied","Data":"800fc303d365379103a8e155cfe33e3ced36d0a3d6814b623663343e62744297"}
Jul 16 20:40:48.671761 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.671623    2576 scope.go:117] "RemoveContainer" containerID="d7fb9366d1d806ed7259d87f57586b936f45361899cab24556259582e70c1eaf"
Jul 16 20:40:48.680635 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.680617    2576 scope.go:117] "RemoveContainer" containerID="d7fb9366d1d806ed7259d87f57586b936f45361899cab24556259582e70c1eaf"
Jul 16 20:40:48.680931 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:40:48.680912    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7fb9366d1d806ed7259d87f57586b936f45361899cab24556259582e70c1eaf\": container with ID starting with d7fb9366d1d806ed7259d87f57586b936f45361899cab24556259582e70c1eaf not found: ID does not exist" containerID="d7fb9366d1d806ed7259d87f57586b936f45361899cab24556259582e70c1eaf"
Jul 16 20:40:48.681030 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.680937    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7fb9366d1d806ed7259d87f57586b936f45361899cab24556259582e70c1eaf"} err="failed to get container status \"d7fb9366d1d806ed7259d87f57586b936f45361899cab24556259582e70c1eaf\": rpc error: code = NotFound desc = could not find container \"d7fb9366d1d806ed7259d87f57586b936f45361899cab24556259582e70c1eaf\": container with ID starting with d7fb9366d1d806ed7259d87f57586b936f45361899cab24556259582e70c1eaf not found: ID does not exist"
Jul 16 20:40:48.694724 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.694649    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"]
Jul 16 20:40:48.702152 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:48.702130    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-955113/test-model-955113-kserve-847cff4d65-rjvcv"]
Jul 16 20:40:50.244384 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:40:50.244354    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90b0c481-5330-4991-b18f-8a02bf87cd42" path="/var/lib/kubelet/pods/90b0c481-5330-4991-b18f-8a02bf87cd42/volumes"
Jul 16 20:41:00.244574 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:00.244543    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"]
Jul 16 20:41:00.245011 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:00.244753    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy" containerID="cri-o://a85f9ed52636506a7c0cedb10bd3ab4c725f95ab467e1913eec3ef0b7a170da6" gracePeriod=30
Jul 16 20:41:05.491175 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.491150    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:41:05.671066 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671034    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-envoy\") pod \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") "
Jul 16 20:41:05.671066 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671068    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-credential-socket\") pod \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") "
Jul 16 20:41:05.671305 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671084    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-workload-socket\") pod \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") "
Jul 16 20:41:05.671305 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671107    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-data\") pod \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") "
Jul 16 20:41:05.671305 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671162    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-podinfo\") pod \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") "
Jul 16 20:41:05.671305 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671191    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szs4p\" (UniqueName: \"kubernetes.io/projected/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-kube-api-access-szs4p\") pod \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") "
Jul 16 20:41:05.671305 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671235    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-token\") pod \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") "
Jul 16 20:41:05.671305 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671277    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-workload-certs\") pod \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") "
Jul 16 20:41:05.671305 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671300    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istiod-ca-cert\") pod \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\" (UID: \"652e8a0f-cb48-42f4-b1ef-8285fe68fe83\") "
Jul 16 20:41:05.671658 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671391    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "652e8a0f-cb48-42f4-b1ef-8285fe68fe83" (UID: "652e8a0f-cb48-42f4-b1ef-8285fe68fe83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:41:05.671658 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671408    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "652e8a0f-cb48-42f4-b1ef-8285fe68fe83" (UID: "652e8a0f-cb48-42f4-b1ef-8285fe68fe83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:41:05.671658 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671649    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-credential-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:41:05.671855 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671668    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-workload-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:41:05.671855 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671738    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "652e8a0f-cb48-42f4-b1ef-8285fe68fe83" (UID: "652e8a0f-cb48-42f4-b1ef-8285fe68fe83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:41:05.671962 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.671859    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "652e8a0f-cb48-42f4-b1ef-8285fe68fe83" (UID: "652e8a0f-cb48-42f4-b1ef-8285fe68fe83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:41:05.672325 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.672297    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-data" (OuterVolumeSpecName: "istio-data") pod "652e8a0f-cb48-42f4-b1ef-8285fe68fe83" (UID: "652e8a0f-cb48-42f4-b1ef-8285fe68fe83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:41:05.673531 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.673509    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "652e8a0f-cb48-42f4-b1ef-8285fe68fe83" (UID: "652e8a0f-cb48-42f4-b1ef-8285fe68fe83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:41:05.673793 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.673769    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-kube-api-access-szs4p" (OuterVolumeSpecName: "kube-api-access-szs4p") pod "652e8a0f-cb48-42f4-b1ef-8285fe68fe83" (UID: "652e8a0f-cb48-42f4-b1ef-8285fe68fe83"). InnerVolumeSpecName "kube-api-access-szs4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:41:05.674051 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.674027    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-token" (OuterVolumeSpecName: "istio-token") pod "652e8a0f-cb48-42f4-b1ef-8285fe68fe83" (UID: "652e8a0f-cb48-42f4-b1ef-8285fe68fe83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:41:05.674118 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.674065    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "652e8a0f-cb48-42f4-b1ef-8285fe68fe83" (UID: "652e8a0f-cb48-42f4-b1ef-8285fe68fe83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 20:41:05.749168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.749140    2576 generic.go:356] "Generic (PLEG): container finished" podID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerID="a85f9ed52636506a7c0cedb10bd3ab4c725f95ab467e1913eec3ef0b7a170da6" exitCode=0
Jul 16 20:41:05.749320 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.749190    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"
Jul 16 20:41:05.749320 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.749228    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" event={"ID":"652e8a0f-cb48-42f4-b1ef-8285fe68fe83","Type":"ContainerDied","Data":"a85f9ed52636506a7c0cedb10bd3ab4c725f95ab467e1913eec3ef0b7a170da6"}
Jul 16 20:41:05.749320 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.749283    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76" event={"ID":"652e8a0f-cb48-42f4-b1ef-8285fe68fe83","Type":"ContainerDied","Data":"c9275b3a268cf9bc8d7dd30be713388be55d74dbd63181987925feca2dde89fa"}
Jul 16 20:41:05.749320 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.749307    2576 scope.go:117] "RemoveContainer" containerID="a85f9ed52636506a7c0cedb10bd3ab4c725f95ab467e1913eec3ef0b7a170da6"
Jul 16 20:41:05.758251 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.758233    2576 scope.go:117] "RemoveContainer" containerID="6c375cd0e1cd753e080d8b19eadcf5821fdcd61b04c702ffb3d8ab71b98c34e3"
Jul 16 20:41:05.766117 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.766100    2576 scope.go:117] "RemoveContainer" containerID="a85f9ed52636506a7c0cedb10bd3ab4c725f95ab467e1913eec3ef0b7a170da6"
Jul 16 20:41:05.766364 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:41:05.766348    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85f9ed52636506a7c0cedb10bd3ab4c725f95ab467e1913eec3ef0b7a170da6\": container with ID starting with a85f9ed52636506a7c0cedb10bd3ab4c725f95ab467e1913eec3ef0b7a170da6 not found: ID does not exist" containerID="a85f9ed52636506a7c0cedb10bd3ab4c725f95ab467e1913eec3ef0b7a170da6"
Jul 16 20:41:05.766414 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.766373    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85f9ed52636506a7c0cedb10bd3ab4c725f95ab467e1913eec3ef0b7a170da6"} err="failed to get container status \"a85f9ed52636506a7c0cedb10bd3ab4c725f95ab467e1913eec3ef0b7a170da6\": rpc error: code = NotFound desc = could not find container \"a85f9ed52636506a7c0cedb10bd3ab4c725f95ab467e1913eec3ef0b7a170da6\": container with ID starting with a85f9ed52636506a7c0cedb10bd3ab4c725f95ab467e1913eec3ef0b7a170da6 not found: ID does not exist"
Jul 16 20:41:05.766414 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.766389    2576 scope.go:117] "RemoveContainer" containerID="6c375cd0e1cd753e080d8b19eadcf5821fdcd61b04c702ffb3d8ab71b98c34e3"
Jul 16 20:41:05.766676 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:41:05.766659    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c375cd0e1cd753e080d8b19eadcf5821fdcd61b04c702ffb3d8ab71b98c34e3\": container with ID starting with 6c375cd0e1cd753e080d8b19eadcf5821fdcd61b04c702ffb3d8ab71b98c34e3 not found: ID does not exist" containerID="6c375cd0e1cd753e080d8b19eadcf5821fdcd61b04c702ffb3d8ab71b98c34e3"
Jul 16 20:41:05.766764 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.766681    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c375cd0e1cd753e080d8b19eadcf5821fdcd61b04c702ffb3d8ab71b98c34e3"} err="failed to get container status \"6c375cd0e1cd753e080d8b19eadcf5821fdcd61b04c702ffb3d8ab71b98c34e3\": rpc error: code = NotFound desc = could not find container \"6c375cd0e1cd753e080d8b19eadcf5821fdcd61b04c702ffb3d8ab71b98c34e3\": container with ID starting with 6c375cd0e1cd753e080d8b19eadcf5821fdcd61b04c702ffb3d8ab71b98c34e3 not found: ID does not exist"
Jul 16 20:41:05.772370 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.772349    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"]
Jul 16 20:41:05.772689 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.772672    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-podinfo\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:41:05.772748 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.772693    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szs4p\" (UniqueName: \"kubernetes.io/projected/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-kube-api-access-szs4p\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:41:05.772748 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.772724    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-token\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:41:05.772748 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.772733    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-workload-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:41:05.772748 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.772741    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istiod-ca-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:41:05.772748 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.772748    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-envoy\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:41:05.772910 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.772756    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652e8a0f-cb48-42f4-b1ef-8285fe68fe83-istio-data\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:41:05.776634 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:05.776615    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-955113-openshift-default-7dd5f845fc-pbf76"]
Jul 16 20:41:06.245647 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:06.245610    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" path="/var/lib/kubelet/pods/652e8a0f-cb48-42f4-b1ef-8285fe68fe83/volumes"
Jul 16 20:41:27.268776 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.268742    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"]
Jul 16 20:41:27.269153 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.269074    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b0c481-5330-4991-b18f-8a02bf87cd42" containerName="main"
Jul 16 20:41:27.269153 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.269084    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b0c481-5330-4991-b18f-8a02bf87cd42" containerName="main"
Jul 16 20:41:27.269153 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.269099    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy"
Jul 16 20:41:27.269153 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.269105    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy"
Jul 16 20:41:27.269153 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.269124    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy"
Jul 16 20:41:27.269153 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.269131    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy"
Jul 16 20:41:27.269363 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.269177    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy"
Jul 16 20:41:27.269363 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.269186    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b0c481-5330-4991-b18f-8a02bf87cd42" containerName="main"
Jul 16 20:41:27.269363 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.269193    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="652e8a0f-cb48-42f4-b1ef-8285fe68fe83" containerName="istio-proxy"
Jul 16 20:41:27.271999 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.271984    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.274534 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.274511    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7cfc57-openshift-default-dockercfg-jqjvg\""
Jul 16 20:41:27.284190 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.284167    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"]
Jul 16 20:41:27.334525 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.334499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-envoy\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.334658 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.334537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687c03a2-ce4b-46e3-90b5-00e993814ed1-istiod-ca-cert\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.334658 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.334558    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-workload-socket\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.334658 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.334575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-credential-socket\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.334658 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.334614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-workload-certs\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.334836 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.334662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-token\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.334836 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.334752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-podinfo\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.334836 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.334776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpwlx\" (UniqueName: \"kubernetes.io/projected/687c03a2-ce4b-46e3-90b5-00e993814ed1-kube-api-access-qpwlx\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.334836 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.334797    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-data\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.435293 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.435253    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpwlx\" (UniqueName: \"kubernetes.io/projected/687c03a2-ce4b-46e3-90b5-00e993814ed1-kube-api-access-qpwlx\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.435293 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.435300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-data\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.435550 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.435341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-envoy\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.435550 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.435373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687c03a2-ce4b-46e3-90b5-00e993814ed1-istiod-ca-cert\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.435550 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.435393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-workload-socket\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.435550 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.435416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-credential-socket\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.435812 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.435614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-workload-certs\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.435812 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.435671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-token\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.435812 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.435765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-podinfo\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.435960 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.435869    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-credential-socket\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.435960 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.435914    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-workload-socket\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.435960 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.435936    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-data\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.436288 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.436161    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-workload-certs\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.436288 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.436274    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687c03a2-ce4b-46e3-90b5-00e993814ed1-istiod-ca-cert\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.437825 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.437800    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-envoy\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.438206 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.438191    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-podinfo\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.443434 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.443403    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-token\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.443657 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.443641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpwlx\" (UniqueName: \"kubernetes.io/projected/687c03a2-ce4b-46e3-90b5-00e993814ed1-kube-api-access-qpwlx\") pod \"e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") " pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.582419 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.582383    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:27.726730 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.726686    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"]
Jul 16 20:41:27.729039 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:41:27.729018    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687c03a2_ce4b_46e3_90b5_00e993814ed1.slice/crio-c1562fd9f78e6b55d4eb5ad553d58579d7483740e72d6480c84a4bdf5348cf20 WatchSource:0}: Error finding container c1562fd9f78e6b55d4eb5ad553d58579d7483740e72d6480c84a4bdf5348cf20: Status 404 returned error can't find the container with id c1562fd9f78e6b55d4eb5ad553d58579d7483740e72d6480c84a4bdf5348cf20
Jul 16 20:41:27.731593 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.731562    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:41:27.731666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.731625    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:41:27.731666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.731658    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:41:27.844438 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.844349    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22" event={"ID":"687c03a2-ce4b-46e3-90b5-00e993814ed1","Type":"ContainerStarted","Data":"6a7a8d297e12679ba718690b0ffd08ec434aa6a59121e0b3cf25d0dd2ff9ed9e"}
Jul 16 20:41:27.844438 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.844404    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22" event={"ID":"687c03a2-ce4b-46e3-90b5-00e993814ed1","Type":"ContainerStarted","Data":"c1562fd9f78e6b55d4eb5ad553d58579d7483740e72d6480c84a4bdf5348cf20"}
Jul 16 20:41:27.866631 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:27.866573    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22" podStartSLOduration=0.866555588 podStartE2EDuration="866.555588ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:27.864552598 +0000 UTC m=+2466.115173363" watchObservedRunningTime="2026-07-16 20:41:27.866555588 +0000 UTC m=+2466.117176347"
Jul 16 20:41:28.582770 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:28.582738    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:28.587355 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:28.587333    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:28.848250 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:28.848173    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:28.849118 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:28.849101    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:34.964954 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:34.964917    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"]
Jul 16 20:41:34.968598 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:34.968576    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:34.971314 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:34.971291    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1dad58-openshift-default-dockercfg-9jwvs\""
Jul 16 20:41:34.982637 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:34.982610    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"]
Jul 16 20:41:35.003151 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.003125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-podinfo\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.003291 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.003162    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-credential-socket\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.003291 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.003182    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-workload-socket\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.003291 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.003234    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-workload-certs\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.003291 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.003285    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-token\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.003584 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.003328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec54a3f7-5254-4473-abb2-bb6477d6b467-istiod-ca-cert\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.003584 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.003369    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfth\" (UniqueName: \"kubernetes.io/projected/ec54a3f7-5254-4473-abb2-bb6477d6b467-kube-api-access-slfth\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.003584 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.003403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-envoy\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.003584 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.003431    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-data\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.104780 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.104742    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfth\" (UniqueName: \"kubernetes.io/projected/ec54a3f7-5254-4473-abb2-bb6477d6b467-kube-api-access-slfth\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.104966 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.104794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-envoy\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.104966 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.104813    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-data\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.104966 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.104853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-podinfo\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.104966 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.104878    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-credential-socket\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.104966 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.104905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-workload-socket\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.104966 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.104930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-workload-certs\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.105274 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.104974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-token\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.105274 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.105024    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec54a3f7-5254-4473-abb2-bb6477d6b467-istiod-ca-cert\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.105432 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.105316    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-credential-socket\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.105432 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.105345    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-data\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.105432 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.105380    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-workload-certs\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.105543 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.105528    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-workload-socket\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.105792 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.105774    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec54a3f7-5254-4473-abb2-bb6477d6b467-istiod-ca-cert\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.107260 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.107241    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-envoy\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.107913 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.107895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-podinfo\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.113322 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.113299    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-token\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.113666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.113646    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfth\" (UniqueName: \"kubernetes.io/projected/ec54a3f7-5254-4473-abb2-bb6477d6b467-kube-api-access-slfth\") pod \"e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") " pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.281020 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.280925    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:35.422556 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.422530    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"]
Jul 16 20:41:35.424719 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:41:35.424669    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec54a3f7_5254_4473_abb2_bb6477d6b467.slice/crio-b33968914fbb3f327ca0a432283dd3b8871458218a9279d7b721f1422bbc3b0c WatchSource:0}: Error finding container b33968914fbb3f327ca0a432283dd3b8871458218a9279d7b721f1422bbc3b0c: Status 404 returned error can't find the container with id b33968914fbb3f327ca0a432283dd3b8871458218a9279d7b721f1422bbc3b0c
Jul 16 20:41:35.427083 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.427051    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:41:35.427184 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.427132    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:41:35.427184 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.427170    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:41:35.880894 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.880861    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" event={"ID":"ec54a3f7-5254-4473-abb2-bb6477d6b467","Type":"ContainerStarted","Data":"cfd3998b752b98294cab25172b75c738008ea9e3797ca9b5b066785b49ab54aa"}
Jul 16 20:41:35.881079 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.880917    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" event={"ID":"ec54a3f7-5254-4473-abb2-bb6477d6b467","Type":"ContainerStarted","Data":"b33968914fbb3f327ca0a432283dd3b8871458218a9279d7b721f1422bbc3b0c"}
Jul 16 20:41:35.901522 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:35.901464    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" podStartSLOduration=1.901443263 podStartE2EDuration="1.901443263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:35.90026403 +0000 UTC m=+2474.150884788" watchObservedRunningTime="2026-07-16 20:41:35.901443263 +0000 UTC m=+2474.152064022"
Jul 16 20:41:36.282189 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:36.282092    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:36.289169 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:36.289059    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:36.886578 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:36.886543    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:36.887533 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:36.887512    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:42.916855 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:42.916826    2576 generic.go:356] "Generic (PLEG): container finished" podID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerID="6a7a8d297e12679ba718690b0ffd08ec434aa6a59121e0b3cf25d0dd2ff9ed9e" exitCode=0
Jul 16 20:41:42.917392 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:42.916901    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22" event={"ID":"687c03a2-ce4b-46e3-90b5-00e993814ed1","Type":"ContainerDied","Data":"6a7a8d297e12679ba718690b0ffd08ec434aa6a59121e0b3cf25d0dd2ff9ed9e"}
Jul 16 20:41:42.917392 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:42.917376    2576 scope.go:117] "RemoveContainer" containerID="6a7a8d297e12679ba718690b0ffd08ec434aa6a59121e0b3cf25d0dd2ff9ed9e"
Jul 16 20:41:42.918446 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:42.918417    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:41:42.918553 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:42.918485    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:41:42.918553 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:42.918520    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:41:43.848658 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:43.848622    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:43.924885 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:43.924852    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22" event={"ID":"687c03a2-ce4b-46e3-90b5-00e993814ed1","Type":"ContainerStarted","Data":"0d83715308ef916ea82f40b33f41496aa59b7b3be0c490265f620d81e33ec925"}
Jul 16 20:41:44.582906 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:44.582865    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:44.584191 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:44.584163    2576 patch_prober.go:28] interesting pod/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 16 20:41:44.584355 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:44.584212    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22" podUID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 16 20:41:45.582993 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:45.582956    2576 patch_prober.go:28] interesting pod/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 16 20:41:45.583390 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:45.583021    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22" podUID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 16 20:41:46.582790 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:46.582751    2576 patch_prober.go:28] interesting pod/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 16 20:41:46.583012 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:46.582802    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22" podUID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 16 20:41:47.583150 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:47.583113    2576 patch_prober.go:28] interesting pod/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 16 20:41:47.583739 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:47.583682    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22" podUID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 16 20:41:48.587636 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:48.587601    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:48.588106 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:48.588049    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:48.588678 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:48.588662    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:41:52.927861 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:52.927821    2576 patch_prober.go:28] interesting pod/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 16 20:41:52.928228 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:52.927876    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 16 20:41:53.972026 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:53.971926    2576 generic.go:356] "Generic (PLEG): container finished" podID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerID="cfd3998b752b98294cab25172b75c738008ea9e3797ca9b5b066785b49ab54aa" exitCode=0
Jul 16 20:41:53.972026 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:53.971980    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" event={"ID":"ec54a3f7-5254-4473-abb2-bb6477d6b467","Type":"ContainerDied","Data":"cfd3998b752b98294cab25172b75c738008ea9e3797ca9b5b066785b49ab54aa"}
Jul 16 20:41:53.972464 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:53.972320    2576 scope.go:117] "RemoveContainer" containerID="cfd3998b752b98294cab25172b75c738008ea9e3797ca9b5b066785b49ab54aa"
Jul 16 20:41:53.973283 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:53.973258    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:41:53.973349 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:53.973322    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:41:53.973385 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:53.973352    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:41:54.979049 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:54.979016    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" event={"ID":"ec54a3f7-5254-4473-abb2-bb6477d6b467","Type":"ContainerStarted","Data":"822ca49cdc65de3474763f50eb31d4bd273e6d6a28a94fc7420fba5f1b8989f8"}
Jul 16 20:41:55.281668 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:55.281570    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:55.281871 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:55.281839    2576 patch_prober.go:28] interesting pod/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 16 20:41:55.281949 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:55.281895    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 16 20:41:56.282083 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:56.282047    2576 patch_prober.go:28] interesting pod/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 16 20:41:56.282495 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:56.282115    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 16 20:41:57.281346 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:57.281307    2576 patch_prober.go:28] interesting pod/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 16 20:41:57.281523 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:57.281363    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 16 20:41:58.282022 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:58.281977    2576 patch_prober.go:28] interesting pod/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 16 20:41:58.282448 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:58.282059    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 16 20:41:59.284805 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:59.284771    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:59.285199 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:59.285119    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:41:59.285744 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:41:59.285727    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:42:16.214220 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:16.214145    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"]
Jul 16 20:42:16.214768 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:16.214381    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22" podUID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerName="istio-proxy" containerID="cri-o://0d83715308ef916ea82f40b33f41496aa59b7b3be0c490265f620d81e33ec925" gracePeriod=30
Jul 16 20:42:18.590228 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:18.590187    2576 patch_prober.go:28] interesting pod/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 20:42:18.590588 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:18.590254    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22" podUID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 20:42:21.464257 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.464233    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:42:21.534549 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.534472    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-podinfo\") pod \"687c03a2-ce4b-46e3-90b5-00e993814ed1\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") "
Jul 16 20:42:21.534549 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.534512    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-workload-certs\") pod \"687c03a2-ce4b-46e3-90b5-00e993814ed1\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") "
Jul 16 20:42:21.534549 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.534534    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-token\") pod \"687c03a2-ce4b-46e3-90b5-00e993814ed1\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") "
Jul 16 20:42:21.534837 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.534567    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-envoy\") pod \"687c03a2-ce4b-46e3-90b5-00e993814ed1\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") "
Jul 16 20:42:21.534837 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.534586    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687c03a2-ce4b-46e3-90b5-00e993814ed1-istiod-ca-cert\") pod \"687c03a2-ce4b-46e3-90b5-00e993814ed1\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") "
Jul 16 20:42:21.534837 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.534604    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-workload-socket\") pod \"687c03a2-ce4b-46e3-90b5-00e993814ed1\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") "
Jul 16 20:42:21.534837 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.534620    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-data\") pod \"687c03a2-ce4b-46e3-90b5-00e993814ed1\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") "
Jul 16 20:42:21.534837 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.534640    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpwlx\" (UniqueName: \"kubernetes.io/projected/687c03a2-ce4b-46e3-90b5-00e993814ed1-kube-api-access-qpwlx\") pod \"687c03a2-ce4b-46e3-90b5-00e993814ed1\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") "
Jul 16 20:42:21.534837 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.534681    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-credential-socket\") pod \"687c03a2-ce4b-46e3-90b5-00e993814ed1\" (UID: \"687c03a2-ce4b-46e3-90b5-00e993814ed1\") "
Jul 16 20:42:21.535160 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.534842    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "687c03a2-ce4b-46e3-90b5-00e993814ed1" (UID: "687c03a2-ce4b-46e3-90b5-00e993814ed1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:42:21.535160 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.534986    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-workload-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:21.535160 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.535133    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "687c03a2-ce4b-46e3-90b5-00e993814ed1" (UID: "687c03a2-ce4b-46e3-90b5-00e993814ed1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:42:21.535485 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.535458    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-data" (OuterVolumeSpecName: "istio-data") pod "687c03a2-ce4b-46e3-90b5-00e993814ed1" (UID: "687c03a2-ce4b-46e3-90b5-00e993814ed1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:42:21.535604 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.535495    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687c03a2-ce4b-46e3-90b5-00e993814ed1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "687c03a2-ce4b-46e3-90b5-00e993814ed1" (UID: "687c03a2-ce4b-46e3-90b5-00e993814ed1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:42:21.535604 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.535570    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "687c03a2-ce4b-46e3-90b5-00e993814ed1" (UID: "687c03a2-ce4b-46e3-90b5-00e993814ed1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:42:21.536803 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.536764    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "687c03a2-ce4b-46e3-90b5-00e993814ed1" (UID: "687c03a2-ce4b-46e3-90b5-00e993814ed1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:42:21.536946 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.536925    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-token" (OuterVolumeSpecName: "istio-token") pod "687c03a2-ce4b-46e3-90b5-00e993814ed1" (UID: "687c03a2-ce4b-46e3-90b5-00e993814ed1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:42:21.537203 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.537190    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "687c03a2-ce4b-46e3-90b5-00e993814ed1" (UID: "687c03a2-ce4b-46e3-90b5-00e993814ed1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 20:42:21.537277 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.537261    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687c03a2-ce4b-46e3-90b5-00e993814ed1-kube-api-access-qpwlx" (OuterVolumeSpecName: "kube-api-access-qpwlx") pod "687c03a2-ce4b-46e3-90b5-00e993814ed1" (UID: "687c03a2-ce4b-46e3-90b5-00e993814ed1"). InnerVolumeSpecName "kube-api-access-qpwlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:42:21.636028 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.635993    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-podinfo\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:21.636028 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.636023    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-token\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:21.636028 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.636032    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-envoy\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:21.636250 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.636039    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687c03a2-ce4b-46e3-90b5-00e993814ed1-istiod-ca-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:21.636250 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.636047    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-workload-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:21.636250 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.636055    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-istio-data\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:21.636250 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.636064    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpwlx\" (UniqueName: \"kubernetes.io/projected/687c03a2-ce4b-46e3-90b5-00e993814ed1-kube-api-access-qpwlx\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:21.636250 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:21.636074    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c03a2-ce4b-46e3-90b5-00e993814ed1-credential-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:22.096321 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:22.096281    2576 generic.go:356] "Generic (PLEG): container finished" podID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerID="0d83715308ef916ea82f40b33f41496aa59b7b3be0c490265f620d81e33ec925" exitCode=0
Jul 16 20:42:22.096509 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:22.096330    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"
Jul 16 20:42:22.096509 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:22.096362    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22" event={"ID":"687c03a2-ce4b-46e3-90b5-00e993814ed1","Type":"ContainerDied","Data":"0d83715308ef916ea82f40b33f41496aa59b7b3be0c490265f620d81e33ec925"}
Jul 16 20:42:22.096509 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:22.096413    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22" event={"ID":"687c03a2-ce4b-46e3-90b5-00e993814ed1","Type":"ContainerDied","Data":"c1562fd9f78e6b55d4eb5ad553d58579d7483740e72d6480c84a4bdf5348cf20"}
Jul 16 20:42:22.096509 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:22.096436    2576 scope.go:117] "RemoveContainer" containerID="0d83715308ef916ea82f40b33f41496aa59b7b3be0c490265f620d81e33ec925"
Jul 16 20:42:22.105671 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:22.105654    2576 scope.go:117] "RemoveContainer" containerID="6a7a8d297e12679ba718690b0ffd08ec434aa6a59121e0b3cf25d0dd2ff9ed9e"
Jul 16 20:42:22.113689 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:22.113669    2576 scope.go:117] "RemoveContainer" containerID="0d83715308ef916ea82f40b33f41496aa59b7b3be0c490265f620d81e33ec925"
Jul 16 20:42:22.114065 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:42:22.114028    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d83715308ef916ea82f40b33f41496aa59b7b3be0c490265f620d81e33ec925\": container with ID starting with 0d83715308ef916ea82f40b33f41496aa59b7b3be0c490265f620d81e33ec925 not found: ID does not exist" containerID="0d83715308ef916ea82f40b33f41496aa59b7b3be0c490265f620d81e33ec925"
Jul 16 20:42:22.114156 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:22.114077    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d83715308ef916ea82f40b33f41496aa59b7b3be0c490265f620d81e33ec925"} err="failed to get container status \"0d83715308ef916ea82f40b33f41496aa59b7b3be0c490265f620d81e33ec925\": rpc error: code = NotFound desc = could not find container \"0d83715308ef916ea82f40b33f41496aa59b7b3be0c490265f620d81e33ec925\": container with ID starting with 0d83715308ef916ea82f40b33f41496aa59b7b3be0c490265f620d81e33ec925 not found: ID does not exist"
Jul 16 20:42:22.114156 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:22.114103    2576 scope.go:117] "RemoveContainer" containerID="6a7a8d297e12679ba718690b0ffd08ec434aa6a59121e0b3cf25d0dd2ff9ed9e"
Jul 16 20:42:22.114362 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:42:22.114344    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7a8d297e12679ba718690b0ffd08ec434aa6a59121e0b3cf25d0dd2ff9ed9e\": container with ID starting with 6a7a8d297e12679ba718690b0ffd08ec434aa6a59121e0b3cf25d0dd2ff9ed9e not found: ID does not exist" containerID="6a7a8d297e12679ba718690b0ffd08ec434aa6a59121e0b3cf25d0dd2ff9ed9e"
Jul 16 20:42:22.114402 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:22.114368    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7a8d297e12679ba718690b0ffd08ec434aa6a59121e0b3cf25d0dd2ff9ed9e"} err="failed to get container status \"6a7a8d297e12679ba718690b0ffd08ec434aa6a59121e0b3cf25d0dd2ff9ed9e\": rpc error: code = NotFound desc = could not find container \"6a7a8d297e12679ba718690b0ffd08ec434aa6a59121e0b3cf25d0dd2ff9ed9e\": container with ID starting with 6a7a8d297e12679ba718690b0ffd08ec434aa6a59121e0b3cf25d0dd2ff9ed9e not found: ID does not exist"
Jul 16 20:42:22.118392 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:22.118371    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"]
Jul 16 20:42:22.122371 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:22.122347    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7cfc57-openshift-default-866b64f8fc-jlx22"]
Jul 16 20:42:22.249851 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:22.249819    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="687c03a2-ce4b-46e3-90b5-00e993814ed1" path="/var/lib/kubelet/pods/687c03a2-ce4b-46e3-90b5-00e993814ed1/volumes"
Jul 16 20:42:42.368125 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:42.368086    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"]
Jul 16 20:42:42.368551 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:42.368296    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerName="istio-proxy" containerID="cri-o://822ca49cdc65de3474763f50eb31d4bd273e6d6a28a94fc7420fba5f1b8989f8" gracePeriod=30
Jul 16 20:42:43.237510 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.237481    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"]
Jul 16 20:42:43.237850 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.237837    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerName="istio-proxy"
Jul 16 20:42:43.237907 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.237852    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerName="istio-proxy"
Jul 16 20:42:43.237907 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.237867    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerName="istio-proxy"
Jul 16 20:42:43.237907 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.237872    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerName="istio-proxy"
Jul 16 20:42:43.238000 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.237925    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerName="istio-proxy"
Jul 16 20:42:43.238000 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.237935    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="687c03a2-ce4b-46e3-90b5-00e993814ed1" containerName="istio-proxy"
Jul 16 20:42:43.240846 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.240826    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.243451 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.243420    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-9b1256-openshift-default-dockercfg-mz6nr\""
Jul 16 20:42:43.252047 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.252021    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"]
Jul 16 20:42:43.341582 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.341515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-workload-certs\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.341816 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.341594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v62zd\" (UniqueName: \"kubernetes.io/projected/e0d58692-9661-4d7e-8b0b-b072136603a4-kube-api-access-v62zd\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.341816 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.341662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-workload-socket\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.341816 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.341732    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-podinfo\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.341816 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.341793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-token\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.341816 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.341815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d58692-9661-4d7e-8b0b-b072136603a4-istiod-ca-cert\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.342102 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.341851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-envoy\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.342102 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.341881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-data\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.342102 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.341910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-credential-socket\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.442627 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.442585    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-envoy\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.442627 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.442633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-data\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.443168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.442663    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-credential-socket\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.443168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.442842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-workload-certs\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.443168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.442883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v62zd\" (UniqueName: \"kubernetes.io/projected/e0d58692-9661-4d7e-8b0b-b072136603a4-kube-api-access-v62zd\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.443168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.442951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-workload-socket\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.443168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.443010    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-podinfo\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.443168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.443070    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-token\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.443168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.443096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d58692-9661-4d7e-8b0b-b072136603a4-istiod-ca-cert\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.443168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.443135    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-data\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.443553 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.443187    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-workload-certs\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.443553 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.443482    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-credential-socket\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.443656 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.443591    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-workload-socket\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.444065 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.444044    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d58692-9661-4d7e-8b0b-b072136603a4-istiod-ca-cert\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.445186 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.445165    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-envoy\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.445767 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.445749    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-podinfo\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.451379 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.451359    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v62zd\" (UniqueName: \"kubernetes.io/projected/e0d58692-9661-4d7e-8b0b-b072136603a4-kube-api-access-v62zd\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.451473 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.451394    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-token\") pod \"e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") " pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.551977 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.551876    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:43.699725 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.699607    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"]
Jul 16 20:42:43.702092 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:42:43.702064    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d58692_9661_4d7e_8b0b_b072136603a4.slice/crio-d4f2a7937a7d428361266b7683ed6d357add3d433a45f029d8ee4def3f105b01 WatchSource:0}: Error finding container d4f2a7937a7d428361266b7683ed6d357add3d433a45f029d8ee4def3f105b01: Status 404 returned error can't find the container with id d4f2a7937a7d428361266b7683ed6d357add3d433a45f029d8ee4def3f105b01
Jul 16 20:42:43.704422 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.704388    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:42:43.704485 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.704465    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:42:43.704522 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:43.704504    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:42:44.198061 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:44.198022    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4" event={"ID":"e0d58692-9661-4d7e-8b0b-b072136603a4","Type":"ContainerStarted","Data":"d1875b744a22d6b520972f26337547639fd3c33bcb97b6aa0c5bf66cd9f2727d"}
Jul 16 20:42:44.198223 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:44.198080    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4" event={"ID":"e0d58692-9661-4d7e-8b0b-b072136603a4","Type":"ContainerStarted","Data":"d4f2a7937a7d428361266b7683ed6d357add3d433a45f029d8ee4def3f105b01"}
Jul 16 20:42:44.219108 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:44.219055    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4" podStartSLOduration=1.219041872 podStartE2EDuration="1.219041872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:42:44.21696857 +0000 UTC m=+2542.467589329" watchObservedRunningTime="2026-07-16 20:42:44.219041872 +0000 UTC m=+2542.469662632"
Jul 16 20:42:44.287435 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:44.287391    2576 patch_prober.go:28] interesting pod/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 20:42:44.287604 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:44.287450    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 20:42:44.552251 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:44.552168    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:44.557323 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:44.557300    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:45.202564 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:45.202532    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:45.203407 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:45.203392    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:42:47.620009 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.619978    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:42:47.680831 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.680737    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-envoy\") pod \"ec54a3f7-5254-4473-abb2-bb6477d6b467\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") "
Jul 16 20:42:47.680831 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.680776    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-token\") pod \"ec54a3f7-5254-4473-abb2-bb6477d6b467\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") "
Jul 16 20:42:47.680831 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.680792    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-data\") pod \"ec54a3f7-5254-4473-abb2-bb6477d6b467\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") "
Jul 16 20:42:47.680831 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.680811    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-workload-socket\") pod \"ec54a3f7-5254-4473-abb2-bb6477d6b467\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") "
Jul 16 20:42:47.680831 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.680833    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-workload-certs\") pod \"ec54a3f7-5254-4473-abb2-bb6477d6b467\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") "
Jul 16 20:42:47.681194 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.680862    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-credential-socket\") pod \"ec54a3f7-5254-4473-abb2-bb6477d6b467\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") "
Jul 16 20:42:47.681194 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.680899    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-podinfo\") pod \"ec54a3f7-5254-4473-abb2-bb6477d6b467\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") "
Jul 16 20:42:47.681194 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.680922    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slfth\" (UniqueName: \"kubernetes.io/projected/ec54a3f7-5254-4473-abb2-bb6477d6b467-kube-api-access-slfth\") pod \"ec54a3f7-5254-4473-abb2-bb6477d6b467\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") "
Jul 16 20:42:47.681194 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.680995    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec54a3f7-5254-4473-abb2-bb6477d6b467-istiod-ca-cert\") pod \"ec54a3f7-5254-4473-abb2-bb6477d6b467\" (UID: \"ec54a3f7-5254-4473-abb2-bb6477d6b467\") "
Jul 16 20:42:47.681194 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.681153    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec54a3f7-5254-4473-abb2-bb6477d6b467" (UID: "ec54a3f7-5254-4473-abb2-bb6477d6b467"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:42:47.681437 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.681203    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec54a3f7-5254-4473-abb2-bb6477d6b467" (UID: "ec54a3f7-5254-4473-abb2-bb6477d6b467"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:42:47.681437 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.681406    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-workload-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:47.681437 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.681427    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-workload-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:47.681572 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.681462    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec54a3f7-5254-4473-abb2-bb6477d6b467" (UID: "ec54a3f7-5254-4473-abb2-bb6477d6b467"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:42:47.681572 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.681486    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec54a3f7-5254-4473-abb2-bb6477d6b467-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec54a3f7-5254-4473-abb2-bb6477d6b467" (UID: "ec54a3f7-5254-4473-abb2-bb6477d6b467"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:42:47.682057 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.682021    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec54a3f7-5254-4473-abb2-bb6477d6b467" (UID: "ec54a3f7-5254-4473-abb2-bb6477d6b467"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:42:47.683293 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.683263    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec54a3f7-5254-4473-abb2-bb6477d6b467" (UID: "ec54a3f7-5254-4473-abb2-bb6477d6b467"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:42:47.683421 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.683354    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec54a3f7-5254-4473-abb2-bb6477d6b467-kube-api-access-slfth" (OuterVolumeSpecName: "kube-api-access-slfth") pod "ec54a3f7-5254-4473-abb2-bb6477d6b467" (UID: "ec54a3f7-5254-4473-abb2-bb6477d6b467"). InnerVolumeSpecName "kube-api-access-slfth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:42:47.683741 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.683692    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec54a3f7-5254-4473-abb2-bb6477d6b467" (UID: "ec54a3f7-5254-4473-abb2-bb6477d6b467"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 20:42:47.683833 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.683801    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec54a3f7-5254-4473-abb2-bb6477d6b467" (UID: "ec54a3f7-5254-4473-abb2-bb6477d6b467"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:42:47.781898 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.781854    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-envoy\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:47.781898 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.781889    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-token\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:47.781898 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.781898    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-data\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:47.781898 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.781907    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec54a3f7-5254-4473-abb2-bb6477d6b467-credential-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:47.782166 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.781917    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec54a3f7-5254-4473-abb2-bb6477d6b467-istio-podinfo\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:47.782166 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.781926    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slfth\" (UniqueName: \"kubernetes.io/projected/ec54a3f7-5254-4473-abb2-bb6477d6b467-kube-api-access-slfth\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:47.782166 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:47.781936    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec54a3f7-5254-4473-abb2-bb6477d6b467-istiod-ca-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:42:48.217779 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:48.217746    2576 generic.go:356] "Generic (PLEG): container finished" podID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerID="822ca49cdc65de3474763f50eb31d4bd273e6d6a28a94fc7420fba5f1b8989f8" exitCode=0
Jul 16 20:42:48.217953 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:48.217796    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"
Jul 16 20:42:48.217953 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:48.217802    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" event={"ID":"ec54a3f7-5254-4473-abb2-bb6477d6b467","Type":"ContainerDied","Data":"822ca49cdc65de3474763f50eb31d4bd273e6d6a28a94fc7420fba5f1b8989f8"}
Jul 16 20:42:48.217953 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:48.217843    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc" event={"ID":"ec54a3f7-5254-4473-abb2-bb6477d6b467","Type":"ContainerDied","Data":"b33968914fbb3f327ca0a432283dd3b8871458218a9279d7b721f1422bbc3b0c"}
Jul 16 20:42:48.217953 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:48.217884    2576 scope.go:117] "RemoveContainer" containerID="822ca49cdc65de3474763f50eb31d4bd273e6d6a28a94fc7420fba5f1b8989f8"
Jul 16 20:42:48.226740 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:48.226683    2576 scope.go:117] "RemoveContainer" containerID="cfd3998b752b98294cab25172b75c738008ea9e3797ca9b5b066785b49ab54aa"
Jul 16 20:42:48.236407 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:48.236382    2576 scope.go:117] "RemoveContainer" containerID="822ca49cdc65de3474763f50eb31d4bd273e6d6a28a94fc7420fba5f1b8989f8"
Jul 16 20:42:48.236857 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:42:48.236836    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822ca49cdc65de3474763f50eb31d4bd273e6d6a28a94fc7420fba5f1b8989f8\": container with ID starting with 822ca49cdc65de3474763f50eb31d4bd273e6d6a28a94fc7420fba5f1b8989f8 not found: ID does not exist" containerID="822ca49cdc65de3474763f50eb31d4bd273e6d6a28a94fc7420fba5f1b8989f8"
Jul 16 20:42:48.236981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:48.236867    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822ca49cdc65de3474763f50eb31d4bd273e6d6a28a94fc7420fba5f1b8989f8"} err="failed to get container status \"822ca49cdc65de3474763f50eb31d4bd273e6d6a28a94fc7420fba5f1b8989f8\": rpc error: code = NotFound desc = could not find container \"822ca49cdc65de3474763f50eb31d4bd273e6d6a28a94fc7420fba5f1b8989f8\": container with ID starting with 822ca49cdc65de3474763f50eb31d4bd273e6d6a28a94fc7420fba5f1b8989f8 not found: ID does not exist"
Jul 16 20:42:48.236981 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:48.236892    2576 scope.go:117] "RemoveContainer" containerID="cfd3998b752b98294cab25172b75c738008ea9e3797ca9b5b066785b49ab54aa"
Jul 16 20:42:48.237357 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:42:48.237323    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd3998b752b98294cab25172b75c738008ea9e3797ca9b5b066785b49ab54aa\": container with ID starting with cfd3998b752b98294cab25172b75c738008ea9e3797ca9b5b066785b49ab54aa not found: ID does not exist" containerID="cfd3998b752b98294cab25172b75c738008ea9e3797ca9b5b066785b49ab54aa"
Jul 16 20:42:48.237467 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:48.237352    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd3998b752b98294cab25172b75c738008ea9e3797ca9b5b066785b49ab54aa"} err="failed to get container status \"cfd3998b752b98294cab25172b75c738008ea9e3797ca9b5b066785b49ab54aa\": rpc error: code = NotFound desc = could not find container \"cfd3998b752b98294cab25172b75c738008ea9e3797ca9b5b066785b49ab54aa\": container with ID starting with cfd3998b752b98294cab25172b75c738008ea9e3797ca9b5b066785b49ab54aa not found: ID does not exist"
Jul 16 20:42:48.244356 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:48.244336    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"]
Jul 16 20:42:48.245116 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:48.245097    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1dad58-openshift-default-6859f8d595-xwdfc"]
Jul 16 20:42:50.244100 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:50.244056    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" path="/var/lib/kubelet/pods/ec54a3f7-5254-4473-abb2-bb6477d6b467/volumes"
Jul 16 20:42:59.267461 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:59.267428    2576 generic.go:356] "Generic (PLEG): container finished" podID="e0d58692-9661-4d7e-8b0b-b072136603a4" containerID="d1875b744a22d6b520972f26337547639fd3c33bcb97b6aa0c5bf66cd9f2727d" exitCode=0
Jul 16 20:42:59.267877 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:59.267472    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4" event={"ID":"e0d58692-9661-4d7e-8b0b-b072136603a4","Type":"ContainerDied","Data":"d1875b744a22d6b520972f26337547639fd3c33bcb97b6aa0c5bf66cd9f2727d"}
Jul 16 20:42:59.267919 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:59.267881    2576 scope.go:117] "RemoveContainer" containerID="d1875b744a22d6b520972f26337547639fd3c33bcb97b6aa0c5bf66cd9f2727d"
Jul 16 20:42:59.268906 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:59.268871    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:42:59.269009 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:59.268958    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:42:59.269077 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:42:59.269003    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:43:00.202990 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:00.202940    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:43:00.276755 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:00.276692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4" event={"ID":"e0d58692-9661-4d7e-8b0b-b072136603a4","Type":"ContainerStarted","Data":"d65d3ea7759d659f031ceea867099018e468d0f0f5820f9ac44de97c22b41774"}
Jul 16 20:43:00.552977 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:00.552881    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:43:00.554174 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:00.554149    2576 patch_prober.go:28] interesting pod/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 16 20:43:00.554290 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:00.554201    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4" podUID="e0d58692-9661-4d7e-8b0b-b072136603a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 16 20:43:01.552578 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:01.552541    2576 patch_prober.go:28] interesting pod/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 16 20:43:01.553006 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:01.552610    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4" podUID="e0d58692-9661-4d7e-8b0b-b072136603a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 16 20:43:02.552597 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:02.552566    2576 patch_prober.go:28] interesting pod/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 16 20:43:02.553003 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:02.552631    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4" podUID="e0d58692-9661-4d7e-8b0b-b072136603a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 16 20:43:03.553167 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:03.553130    2576 patch_prober.go:28] interesting pod/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 16 20:43:03.553572 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:03.553187    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4" podUID="e0d58692-9661-4d7e-8b0b-b072136603a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 16 20:43:04.556540 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:04.556512    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:43:04.556962 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:04.556803    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:43:04.557444 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:04.557423    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:43:14.362131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.362099    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"]
Jul 16 20:43:14.362493 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.362444    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerName="istio-proxy"
Jul 16 20:43:14.362493 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.362457    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerName="istio-proxy"
Jul 16 20:43:14.362493 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.362478    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerName="istio-proxy"
Jul 16 20:43:14.362493 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.362485    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerName="istio-proxy"
Jul 16 20:43:14.362624 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.362541    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerName="istio-proxy"
Jul 16 20:43:14.364642 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.364625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.367380 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.367357    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9b1256\"/\"kube-root-ca.crt\""
Jul 16 20:43:14.367380 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.367370    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9b1256\"/\"openshift-service-ca.crt\""
Jul 16 20:43:14.368637 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.368623    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9b1256\"/\"default-dockercfg-r6tbv\""
Jul 16 20:43:14.368719 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.368648    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9b1256\"/\"auth-test-model-9b1256-kserve-self-signed-certs\""
Jul 16 20:43:14.379149 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.379129    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"]
Jul 16 20:43:14.420300 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.420269    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-model-cache\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.420480 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.420308    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-home\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.420480 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.420377    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-dshm\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.420480 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.420394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20276d5e-bbeb-4e0d-b875-8688b636d282-tls-certs\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.420480 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.420444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-tmp-dir\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.420480 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.420463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddfhm\" (UniqueName: \"kubernetes.io/projected/20276d5e-bbeb-4e0d-b875-8688b636d282-kube-api-access-ddfhm\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.521815 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.521779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-model-cache\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.522007 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.521827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-home\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.522007 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.521881    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-dshm\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.522007 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.521905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20276d5e-bbeb-4e0d-b875-8688b636d282-tls-certs\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.522007 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.521938    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-tmp-dir\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.522007 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.521961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddfhm\" (UniqueName: \"kubernetes.io/projected/20276d5e-bbeb-4e0d-b875-8688b636d282-kube-api-access-ddfhm\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.522274 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.522228    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-model-cache\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.522340 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.522296    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-home\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.522427 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.522328    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-tmp-dir\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.524204 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.524182    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-dshm\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.524455 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.524441    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20276d5e-bbeb-4e0d-b875-8688b636d282-tls-certs\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.536863 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.536836    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddfhm\" (UniqueName: \"kubernetes.io/projected/20276d5e-bbeb-4e0d-b875-8688b636d282-kube-api-access-ddfhm\") pod \"auth-test-model-9b1256-kserve-74f6b88b7f-2btbx\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") " pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.680926 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.680851    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:14.809831 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:14.809805    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"]
Jul 16 20:43:14.811690 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:43:14.811662    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20276d5e_bbeb_4e0d_b875_8688b636d282.slice/crio-b9ef2a0960eba97d02d33cdfeb8598396da62e1d50581d22112599103f987e42 WatchSource:0}: Error finding container b9ef2a0960eba97d02d33cdfeb8598396da62e1d50581d22112599103f987e42: Status 404 returned error can't find the container with id b9ef2a0960eba97d02d33cdfeb8598396da62e1d50581d22112599103f987e42
Jul 16 20:43:15.343131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:15.343087    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx" event={"ID":"20276d5e-bbeb-4e0d-b875-8688b636d282","Type":"ContainerStarted","Data":"5e53a520523d6e1ceaaf8cb29be2fe9a97a936586ee5ee4d10fccf3f839e932d"}
Jul 16 20:43:15.343131 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:15.343139    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx" event={"ID":"20276d5e-bbeb-4e0d-b875-8688b636d282","Type":"ContainerStarted","Data":"b9ef2a0960eba97d02d33cdfeb8598396da62e1d50581d22112599103f987e42"}
Jul 16 20:43:15.364869 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:15.364819    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx" podStartSLOduration=1.364804551 podStartE2EDuration="1.364804551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:43:15.36379181 +0000 UTC m=+2573.614412571" watchObservedRunningTime="2026-07-16 20:43:15.364804551 +0000 UTC m=+2573.615425309"
Jul 16 20:43:24.681453 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:24.681425    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:24.681928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:24.681667    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:24.693756 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:24.693734    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:43:25.397835 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:43:25.397806    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:46:03.013742 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:03.013692    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"]
Jul 16 20:46:16.620908 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.620876    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"]
Jul 16 20:46:16.621271 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.621252    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec54a3f7-5254-4473-abb2-bb6477d6b467" containerName="istio-proxy"
Jul 16 20:46:16.624179 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.624162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.626558 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.626533    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9b1256\"/\"sub-test-model-9b1256-kserve-self-signed-certs\""
Jul 16 20:46:16.633606 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.633585    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"]
Jul 16 20:46:16.772086 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.772054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-tmp-dir\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.772245 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.772105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf696\" (UniqueName: \"kubernetes.io/projected/1467a77b-9db8-4839-8507-3072777bb1c0-kube-api-access-vf696\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.772245 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.772154    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-home\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.772245 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.772181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1467a77b-9db8-4839-8507-3072777bb1c0-tls-certs\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.772245 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.772201    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-model-cache\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.772387 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.772259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-dshm\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.873582 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.873492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-home\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.873582 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.873549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1467a77b-9db8-4839-8507-3072777bb1c0-tls-certs\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.873582 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.873581    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-model-cache\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.873888 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.873612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-dshm\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.873888 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.873727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-tmp-dir\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.873888 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.873795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf696\" (UniqueName: \"kubernetes.io/projected/1467a77b-9db8-4839-8507-3072777bb1c0-kube-api-access-vf696\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.874011 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.873968    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-home\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.874154 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.874128    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-tmp-dir\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.874230 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.874215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-model-cache\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.876084 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.876061    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-dshm\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.876286 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.876269    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1467a77b-9db8-4839-8507-3072777bb1c0-tls-certs\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.882584 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.882561    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf696\" (UniqueName: \"kubernetes.io/projected/1467a77b-9db8-4839-8507-3072777bb1c0-kube-api-access-vf696\") pod \"sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") " pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:16.933759 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:16.933726    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:17.069162 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:17.069131    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"]
Jul 16 20:46:17.072334 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:46:17.072303    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1467a77b_9db8_4839_8507_3072777bb1c0.slice/crio-25c0bd60f2666d802ec744bbfb0247b204908cb2540392cddab7329c1377eff0 WatchSource:0}: Error finding container 25c0bd60f2666d802ec744bbfb0247b204908cb2540392cddab7329c1377eff0: Status 404 returned error can't find the container with id 25c0bd60f2666d802ec744bbfb0247b204908cb2540392cddab7329c1377eff0
Jul 16 20:46:17.116902 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:17.116874    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw" event={"ID":"1467a77b-9db8-4839-8507-3072777bb1c0","Type":"ContainerStarted","Data":"25c0bd60f2666d802ec744bbfb0247b204908cb2540392cddab7329c1377eff0"}
Jul 16 20:46:18.034372 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.034335    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx" podUID="20276d5e-bbeb-4e0d-b875-8688b636d282" containerName="main" containerID="cri-o://5e53a520523d6e1ceaaf8cb29be2fe9a97a936586ee5ee4d10fccf3f839e932d" gracePeriod=45
Jul 16 20:46:18.122545 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.122509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw" event={"ID":"1467a77b-9db8-4839-8507-3072777bb1c0","Type":"ContainerStarted","Data":"e914c4859bf4a907a89da7e26fe15ed0b46f13e25076d4d9dd723f8064c83ac2"}
Jul 16 20:46:18.141388 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.141329    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw" podStartSLOduration=2.141306867 podStartE2EDuration="2.141306867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:46:18.138777371 +0000 UTC m=+2756.389398133" watchObservedRunningTime="2026-07-16 20:46:18.141306867 +0000 UTC m=+2756.391927628"
Jul 16 20:46:18.292808 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.292756    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:46:18.388914 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.388881    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-model-cache\") pod \"20276d5e-bbeb-4e0d-b875-8688b636d282\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") "
Jul 16 20:46:18.389089 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.388968    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-home\") pod \"20276d5e-bbeb-4e0d-b875-8688b636d282\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") "
Jul 16 20:46:18.389089 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.388997    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-tmp-dir\") pod \"20276d5e-bbeb-4e0d-b875-8688b636d282\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") "
Jul 16 20:46:18.389089 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.389068    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20276d5e-bbeb-4e0d-b875-8688b636d282-tls-certs\") pod \"20276d5e-bbeb-4e0d-b875-8688b636d282\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") "
Jul 16 20:46:18.389254 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.389098    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-dshm\") pod \"20276d5e-bbeb-4e0d-b875-8688b636d282\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") "
Jul 16 20:46:18.389254 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.389125    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddfhm\" (UniqueName: \"kubernetes.io/projected/20276d5e-bbeb-4e0d-b875-8688b636d282-kube-api-access-ddfhm\") pod \"20276d5e-bbeb-4e0d-b875-8688b636d282\" (UID: \"20276d5e-bbeb-4e0d-b875-8688b636d282\") "
Jul 16 20:46:18.389254 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.389169    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-model-cache" (OuterVolumeSpecName: "model-cache") pod "20276d5e-bbeb-4e0d-b875-8688b636d282" (UID: "20276d5e-bbeb-4e0d-b875-8688b636d282"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:46:18.389382 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.389249    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-home" (OuterVolumeSpecName: "home") pod "20276d5e-bbeb-4e0d-b875-8688b636d282" (UID: "20276d5e-bbeb-4e0d-b875-8688b636d282"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:46:18.389382 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.389335    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20276d5e-bbeb-4e0d-b875-8688b636d282" (UID: "20276d5e-bbeb-4e0d-b875-8688b636d282"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:46:18.389505 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.389487    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-home\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:46:18.389559 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.389508    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-tmp-dir\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:46:18.389559 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.389518    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-model-cache\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:46:18.391503 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.391476    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-dshm" (OuterVolumeSpecName: "dshm") pod "20276d5e-bbeb-4e0d-b875-8688b636d282" (UID: "20276d5e-bbeb-4e0d-b875-8688b636d282"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:46:18.391618 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.391540    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20276d5e-bbeb-4e0d-b875-8688b636d282-kube-api-access-ddfhm" (OuterVolumeSpecName: "kube-api-access-ddfhm") pod "20276d5e-bbeb-4e0d-b875-8688b636d282" (UID: "20276d5e-bbeb-4e0d-b875-8688b636d282"). InnerVolumeSpecName "kube-api-access-ddfhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:46:18.391618 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.391590    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20276d5e-bbeb-4e0d-b875-8688b636d282-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20276d5e-bbeb-4e0d-b875-8688b636d282" (UID: "20276d5e-bbeb-4e0d-b875-8688b636d282"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:46:18.490972 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.490937    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20276d5e-bbeb-4e0d-b875-8688b636d282-tls-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:46:18.490972 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.490966    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20276d5e-bbeb-4e0d-b875-8688b636d282-dshm\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:46:18.491169 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:18.490975    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddfhm\" (UniqueName: \"kubernetes.io/projected/20276d5e-bbeb-4e0d-b875-8688b636d282-kube-api-access-ddfhm\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:46:19.128455 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:19.128419    2576 generic.go:356] "Generic (PLEG): container finished" podID="20276d5e-bbeb-4e0d-b875-8688b636d282" containerID="5e53a520523d6e1ceaaf8cb29be2fe9a97a936586ee5ee4d10fccf3f839e932d" exitCode=0
Jul 16 20:46:19.128895 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:19.128459    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"
Jul 16 20:46:19.128895 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:19.128503    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx" event={"ID":"20276d5e-bbeb-4e0d-b875-8688b636d282","Type":"ContainerDied","Data":"5e53a520523d6e1ceaaf8cb29be2fe9a97a936586ee5ee4d10fccf3f839e932d"}
Jul 16 20:46:19.128895 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:19.128546    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx" event={"ID":"20276d5e-bbeb-4e0d-b875-8688b636d282","Type":"ContainerDied","Data":"b9ef2a0960eba97d02d33cdfeb8598396da62e1d50581d22112599103f987e42"}
Jul 16 20:46:19.128895 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:19.128563    2576 scope.go:117] "RemoveContainer" containerID="5e53a520523d6e1ceaaf8cb29be2fe9a97a936586ee5ee4d10fccf3f839e932d"
Jul 16 20:46:19.137865 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:19.137850    2576 scope.go:117] "RemoveContainer" containerID="5e53a520523d6e1ceaaf8cb29be2fe9a97a936586ee5ee4d10fccf3f839e932d"
Jul 16 20:46:19.138139 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:46:19.138121    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e53a520523d6e1ceaaf8cb29be2fe9a97a936586ee5ee4d10fccf3f839e932d\": container with ID starting with 5e53a520523d6e1ceaaf8cb29be2fe9a97a936586ee5ee4d10fccf3f839e932d not found: ID does not exist" containerID="5e53a520523d6e1ceaaf8cb29be2fe9a97a936586ee5ee4d10fccf3f839e932d"
Jul 16 20:46:19.138193 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:19.138147    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e53a520523d6e1ceaaf8cb29be2fe9a97a936586ee5ee4d10fccf3f839e932d"} err="failed to get container status \"5e53a520523d6e1ceaaf8cb29be2fe9a97a936586ee5ee4d10fccf3f839e932d\": rpc error: code = NotFound desc = could not find container \"5e53a520523d6e1ceaaf8cb29be2fe9a97a936586ee5ee4d10fccf3f839e932d\": container with ID starting with 5e53a520523d6e1ceaaf8cb29be2fe9a97a936586ee5ee4d10fccf3f839e932d not found: ID does not exist"
Jul 16 20:46:19.149767 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:19.149739    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"]
Jul 16 20:46:19.151788 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:19.151767    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/auth-test-model-9b1256-kserve-74f6b88b7f-2btbx"]
Jul 16 20:46:20.244225 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:20.244194    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20276d5e-bbeb-4e0d-b875-8688b636d282" path="/var/lib/kubelet/pods/20276d5e-bbeb-4e0d-b875-8688b636d282/volumes"
Jul 16 20:46:26.934405 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:26.934359    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:26.934928 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:26.934430    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:26.946962 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:26.946937    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:46:27.182808 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:46:27.182776    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:47:51.823372 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:51.823339    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"]
Jul 16 20:47:59.016670 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.016626    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"]
Jul 16 20:47:59.019107 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.017119    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20276d5e-bbeb-4e0d-b875-8688b636d282" containerName="main"
Jul 16 20:47:59.019107 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.017134    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="20276d5e-bbeb-4e0d-b875-8688b636d282" containerName="main"
Jul 16 20:47:59.019107 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.017202    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="20276d5e-bbeb-4e0d-b875-8688b636d282" containerName="main"
Jul 16 20:47:59.019953 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.019935    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.022435 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.022418    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9b1256\"/\"rate-test-model-9b1256-kserve-self-signed-certs\""
Jul 16 20:47:59.029342 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.029322    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"]
Jul 16 20:47:59.088756 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.088720    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-tmp-dir\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.088756 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.088757    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-dshm\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.089008 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.088775    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-home\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.089008 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.088876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-model-cache\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.089008 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.088975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c38418e-c685-468e-b313-c2f3b628be15-tls-certs\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.089008 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.089002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz9zb\" (UniqueName: \"kubernetes.io/projected/4c38418e-c685-468e-b313-c2f3b628be15-kube-api-access-nz9zb\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.190147 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.190114    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-model-cache\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.190325 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.190166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c38418e-c685-468e-b313-c2f3b628be15-tls-certs\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.190325 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.190187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz9zb\" (UniqueName: \"kubernetes.io/projected/4c38418e-c685-468e-b313-c2f3b628be15-kube-api-access-nz9zb\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.190325 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.190234    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-tmp-dir\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.190325 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.190251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-dshm\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.190325 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.190267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-home\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.190590 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.190552    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-model-cache\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.190668 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.190649    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-tmp-dir\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.190743 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.190692    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-home\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.192584 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.192567    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-dshm\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.192905 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.192887    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c38418e-c685-468e-b313-c2f3b628be15-tls-certs\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.199479 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.199456    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz9zb\" (UniqueName: \"kubernetes.io/projected/4c38418e-c685-468e-b313-c2f3b628be15-kube-api-access-nz9zb\") pod \"rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") " pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.329988 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.329952    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:47:59.464936 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.464909    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"]
Jul 16 20:47:59.467121 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:47:59.467089    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c38418e_c685_468e_b313_c2f3b628be15.slice/crio-786bfb8378601d409857b8a98a690aa8fb73138cc4521d457e388f9c9c9d31b0 WatchSource:0}: Error finding container 786bfb8378601d409857b8a98a690aa8fb73138cc4521d457e388f9c9c9d31b0: Status 404 returned error can't find the container with id 786bfb8378601d409857b8a98a690aa8fb73138cc4521d457e388f9c9c9d31b0
Jul 16 20:47:59.580338 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.580255    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s" event={"ID":"4c38418e-c685-468e-b313-c2f3b628be15","Type":"ContainerStarted","Data":"9d646373b0d66f84d86c84661f3baef958f8bc27a8aee5bdc78f2db347ba4fba"}
Jul 16 20:47:59.580338 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.580301    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s" event={"ID":"4c38418e-c685-468e-b313-c2f3b628be15","Type":"ContainerStarted","Data":"786bfb8378601d409857b8a98a690aa8fb73138cc4521d457e388f9c9c9d31b0"}
Jul 16 20:47:59.598620 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:47:59.598573    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s" podStartSLOduration=1.5985578089999999 podStartE2EDuration="1.598557809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:47:59.597591097 +0000 UTC m=+2857.848211857" watchObservedRunningTime="2026-07-16 20:47:59.598557809 +0000 UTC m=+2857.849178568"
Jul 16 20:48:06.843395 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:06.843328    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw" podUID="1467a77b-9db8-4839-8507-3072777bb1c0" containerName="main" containerID="cri-o://e914c4859bf4a907a89da7e26fe15ed0b46f13e25076d4d9dd723f8064c83ac2" gracePeriod=45
Jul 16 20:48:07.092925 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.092894    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:48:07.164233 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.164159    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf696\" (UniqueName: \"kubernetes.io/projected/1467a77b-9db8-4839-8507-3072777bb1c0-kube-api-access-vf696\") pod \"1467a77b-9db8-4839-8507-3072777bb1c0\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") "
Jul 16 20:48:07.164233 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.164194    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-dshm\") pod \"1467a77b-9db8-4839-8507-3072777bb1c0\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") "
Jul 16 20:48:07.164443 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.164254    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-tmp-dir\") pod \"1467a77b-9db8-4839-8507-3072777bb1c0\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") "
Jul 16 20:48:07.164443 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.164277    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1467a77b-9db8-4839-8507-3072777bb1c0-tls-certs\") pod \"1467a77b-9db8-4839-8507-3072777bb1c0\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") "
Jul 16 20:48:07.164443 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.164305    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-model-cache\") pod \"1467a77b-9db8-4839-8507-3072777bb1c0\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") "
Jul 16 20:48:07.164443 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.164331    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-home\") pod \"1467a77b-9db8-4839-8507-3072777bb1c0\" (UID: \"1467a77b-9db8-4839-8507-3072777bb1c0\") "
Jul 16 20:48:07.164653 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.164575    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1467a77b-9db8-4839-8507-3072777bb1c0" (UID: "1467a77b-9db8-4839-8507-3072777bb1c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:48:07.164653 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.164611    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "1467a77b-9db8-4839-8507-3072777bb1c0" (UID: "1467a77b-9db8-4839-8507-3072777bb1c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:48:07.164653 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.164626    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-home" (OuterVolumeSpecName: "home") pod "1467a77b-9db8-4839-8507-3072777bb1c0" (UID: "1467a77b-9db8-4839-8507-3072777bb1c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:48:07.164836 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.164751    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-model-cache\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:48:07.164836 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.164771    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-home\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:48:07.164836 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.164786    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-tmp-dir\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:48:07.166519 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.166498    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-dshm" (OuterVolumeSpecName: "dshm") pod "1467a77b-9db8-4839-8507-3072777bb1c0" (UID: "1467a77b-9db8-4839-8507-3072777bb1c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:48:07.166605 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.166592    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1467a77b-9db8-4839-8507-3072777bb1c0-kube-api-access-vf696" (OuterVolumeSpecName: "kube-api-access-vf696") pod "1467a77b-9db8-4839-8507-3072777bb1c0" (UID: "1467a77b-9db8-4839-8507-3072777bb1c0"). InnerVolumeSpecName "kube-api-access-vf696". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:48:07.166646 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.166626    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1467a77b-9db8-4839-8507-3072777bb1c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1467a77b-9db8-4839-8507-3072777bb1c0" (UID: "1467a77b-9db8-4839-8507-3072777bb1c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:48:07.265516 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.265488    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf696\" (UniqueName: \"kubernetes.io/projected/1467a77b-9db8-4839-8507-3072777bb1c0-kube-api-access-vf696\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:48:07.265516 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.265513    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1467a77b-9db8-4839-8507-3072777bb1c0-dshm\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:48:07.265668 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.265524    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1467a77b-9db8-4839-8507-3072777bb1c0-tls-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:48:07.619906 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.619870    2576 generic.go:356] "Generic (PLEG): container finished" podID="1467a77b-9db8-4839-8507-3072777bb1c0" containerID="e914c4859bf4a907a89da7e26fe15ed0b46f13e25076d4d9dd723f8064c83ac2" exitCode=0
Jul 16 20:48:07.619906 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.619909    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw" event={"ID":"1467a77b-9db8-4839-8507-3072777bb1c0","Type":"ContainerDied","Data":"e914c4859bf4a907a89da7e26fe15ed0b46f13e25076d4d9dd723f8064c83ac2"}
Jul 16 20:48:07.620121 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.619914    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"
Jul 16 20:48:07.620121 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.619938    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw" event={"ID":"1467a77b-9db8-4839-8507-3072777bb1c0","Type":"ContainerDied","Data":"25c0bd60f2666d802ec744bbfb0247b204908cb2540392cddab7329c1377eff0"}
Jul 16 20:48:07.620121 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.619960    2576 scope.go:117] "RemoveContainer" containerID="e914c4859bf4a907a89da7e26fe15ed0b46f13e25076d4d9dd723f8064c83ac2"
Jul 16 20:48:07.628560 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.628545    2576 scope.go:117] "RemoveContainer" containerID="e914c4859bf4a907a89da7e26fe15ed0b46f13e25076d4d9dd723f8064c83ac2"
Jul 16 20:48:07.628835 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:48:07.628818    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e914c4859bf4a907a89da7e26fe15ed0b46f13e25076d4d9dd723f8064c83ac2\": container with ID starting with e914c4859bf4a907a89da7e26fe15ed0b46f13e25076d4d9dd723f8064c83ac2 not found: ID does not exist" containerID="e914c4859bf4a907a89da7e26fe15ed0b46f13e25076d4d9dd723f8064c83ac2"
Jul 16 20:48:07.628889 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.628843    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e914c4859bf4a907a89da7e26fe15ed0b46f13e25076d4d9dd723f8064c83ac2"} err="failed to get container status \"e914c4859bf4a907a89da7e26fe15ed0b46f13e25076d4d9dd723f8064c83ac2\": rpc error: code = NotFound desc = could not find container \"e914c4859bf4a907a89da7e26fe15ed0b46f13e25076d4d9dd723f8064c83ac2\": container with ID starting with e914c4859bf4a907a89da7e26fe15ed0b46f13e25076d4d9dd723f8064c83ac2 not found: ID does not exist"
Jul 16 20:48:07.639479 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.639458    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"]
Jul 16 20:48:07.642645 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:07.642622    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/sub-test-model-9b1256-kserve-6f7fbf9d49-fx6rw"]
Jul 16 20:48:08.244886 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:08.244852    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1467a77b-9db8-4839-8507-3072777bb1c0" path="/var/lib/kubelet/pods/1467a77b-9db8-4839-8507-3072777bb1c0/volumes"
Jul 16 20:48:09.331136 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:09.331106    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:48:09.331497 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:09.331162    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:48:09.343880 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:09.343855    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:48:09.640858 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:48:09.640783    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:49:13.476445 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:13.476412    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"]
Jul 16 20:49:18.507754 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:18.507723    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7357-139849673670832"]
Jul 16 20:49:18.508119 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:18.508051    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1467a77b-9db8-4839-8507-3072777bb1c0" containerName="main"
Jul 16 20:49:18.508119 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:18.508063    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1467a77b-9db8-4839-8507-3072777bb1c0" containerName="main"
Jul 16 20:49:18.508197 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:18.508125    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1467a77b-9db8-4839-8507-3072777bb1c0" containerName="main"
Jul 16 20:49:18.511298 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:18.511282    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673670832"
Jul 16 20:49:18.513830 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:18.513778    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lhm6n\""
Jul 16 20:49:18.515076 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:18.515055    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7357-139849673670832"]
Jul 16 20:49:18.599782 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:18.599742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jq4z\" (UniqueName: \"kubernetes.io/projected/4a5e35c8-1a18-40be-b3ee-3d515a62c1d3-kube-api-access-5jq4z\") pod \"test-curl-7357-139849673670832\" (UID: \"4a5e35c8-1a18-40be-b3ee-3d515a62c1d3\") " pod="opendatahub/test-curl-7357-139849673670832"
Jul 16 20:49:18.700662 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:18.700628    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jq4z\" (UniqueName: \"kubernetes.io/projected/4a5e35c8-1a18-40be-b3ee-3d515a62c1d3-kube-api-access-5jq4z\") pod \"test-curl-7357-139849673670832\" (UID: \"4a5e35c8-1a18-40be-b3ee-3d515a62c1d3\") " pod="opendatahub/test-curl-7357-139849673670832"
Jul 16 20:49:18.710591 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:18.710566    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jq4z\" (UniqueName: \"kubernetes.io/projected/4a5e35c8-1a18-40be-b3ee-3d515a62c1d3-kube-api-access-5jq4z\") pod \"test-curl-7357-139849673670832\" (UID: \"4a5e35c8-1a18-40be-b3ee-3d515a62c1d3\") " pod="opendatahub/test-curl-7357-139849673670832"
Jul 16 20:49:18.822075 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:18.822040    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673670832"
Jul 16 20:49:18.955055 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:18.955031    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7357-139849673670832"]
Jul 16 20:49:18.957669 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:49:18.957641    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a5e35c8_1a18_40be_b3ee_3d515a62c1d3.slice/crio-522093adfa4a852e3adfd0620c33e78637b67ca1845a291f8b484cb769c76b67 WatchSource:0}: Error finding container 522093adfa4a852e3adfd0620c33e78637b67ca1845a291f8b484cb769c76b67: Status 404 returned error can't find the container with id 522093adfa4a852e3adfd0620c33e78637b67ca1845a291f8b484cb769c76b67
Jul 16 20:49:18.959116 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:18.959097    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 20:49:19.942181 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:19.942092    2576 generic.go:356] "Generic (PLEG): container finished" podID="4a5e35c8-1a18-40be-b3ee-3d515a62c1d3" containerID="5f15fe10694f2a6657750848055d2a539f805b3acecb890dd9af00d8b23d10ed" exitCode=0
Jul 16 20:49:19.942181 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:19.942135    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7357-139849673670832" event={"ID":"4a5e35c8-1a18-40be-b3ee-3d515a62c1d3","Type":"ContainerDied","Data":"5f15fe10694f2a6657750848055d2a539f805b3acecb890dd9af00d8b23d10ed"}
Jul 16 20:49:19.942181 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:19.942179    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7357-139849673670832" event={"ID":"4a5e35c8-1a18-40be-b3ee-3d515a62c1d3","Type":"ContainerStarted","Data":"522093adfa4a852e3adfd0620c33e78637b67ca1845a291f8b484cb769c76b67"}
Jul 16 20:49:21.072447 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.072418    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673670832"
Jul 16 20:49:21.109577 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.109549    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7357-139849673670832_4a5e35c8-1a18-40be-b3ee-3d515a62c1d3/test-curl-7357-139849673670832/0.log"
Jul 16 20:49:21.135623 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.135590    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7357-139849673670832"]
Jul 16 20:49:21.139480 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.139458    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7357-139849673670832"]
Jul 16 20:49:21.223049 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.222956    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jq4z\" (UniqueName: \"kubernetes.io/projected/4a5e35c8-1a18-40be-b3ee-3d515a62c1d3-kube-api-access-5jq4z\") pod \"4a5e35c8-1a18-40be-b3ee-3d515a62c1d3\" (UID: \"4a5e35c8-1a18-40be-b3ee-3d515a62c1d3\") "
Jul 16 20:49:21.225291 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.225260    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5e35c8-1a18-40be-b3ee-3d515a62c1d3-kube-api-access-5jq4z" (OuterVolumeSpecName: "kube-api-access-5jq4z") pod "4a5e35c8-1a18-40be-b3ee-3d515a62c1d3" (UID: "4a5e35c8-1a18-40be-b3ee-3d515a62c1d3"). InnerVolumeSpecName "kube-api-access-5jq4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:49:21.324214 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.324179    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jq4z\" (UniqueName: \"kubernetes.io/projected/4a5e35c8-1a18-40be-b3ee-3d515a62c1d3-kube-api-access-5jq4z\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:49:21.386308 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.386277    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7357-139849673757008"]
Jul 16 20:49:21.386602 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.386590    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a5e35c8-1a18-40be-b3ee-3d515a62c1d3" containerName="test-curl-7357-139849673670832"
Jul 16 20:49:21.386644 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.386604    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a5e35c8-1a18-40be-b3ee-3d515a62c1d3" containerName="test-curl-7357-139849673670832"
Jul 16 20:49:21.386679 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.386672    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a5e35c8-1a18-40be-b3ee-3d515a62c1d3" containerName="test-curl-7357-139849673670832"
Jul 16 20:49:21.389617 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.389603    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:21.401388 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.401365    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7357-139849673757008"]
Jul 16 20:49:21.526152 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.526068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p69b\" (UniqueName: \"kubernetes.io/projected/c28ff645-a7de-4ae8-9b6f-e36b3c758109-kube-api-access-5p69b\") pod \"test-curl-7357-139849673757008\" (UID: \"c28ff645-a7de-4ae8-9b6f-e36b3c758109\") " pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:21.627300 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.627266    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p69b\" (UniqueName: \"kubernetes.io/projected/c28ff645-a7de-4ae8-9b6f-e36b3c758109-kube-api-access-5p69b\") pod \"test-curl-7357-139849673757008\" (UID: \"c28ff645-a7de-4ae8-9b6f-e36b3c758109\") " pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:21.636191 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.636172    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p69b\" (UniqueName: \"kubernetes.io/projected/c28ff645-a7de-4ae8-9b6f-e36b3c758109-kube-api-access-5p69b\") pod \"test-curl-7357-139849673757008\" (UID: \"c28ff645-a7de-4ae8-9b6f-e36b3c758109\") " pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:21.698189 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.698157    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:21.829534 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.829504    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7357-139849673757008"]
Jul 16 20:49:21.832533 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:49:21.832502    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc28ff645_a7de_4ae8_9b6f_e36b3c758109.slice/crio-df670ef52c3f3e5b2a2dece2f277a696815059ee0b9c8d1a36bedd46dffae2eb WatchSource:0}: Error finding container df670ef52c3f3e5b2a2dece2f277a696815059ee0b9c8d1a36bedd46dffae2eb: Status 404 returned error can't find the container with id df670ef52c3f3e5b2a2dece2f277a696815059ee0b9c8d1a36bedd46dffae2eb
Jul 16 20:49:21.952010 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.951968    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7357-139849673757008" event={"ID":"c28ff645-a7de-4ae8-9b6f-e36b3c758109","Type":"ContainerStarted","Data":"df670ef52c3f3e5b2a2dece2f277a696815059ee0b9c8d1a36bedd46dffae2eb"}
Jul 16 20:49:21.953221 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.953201    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="522093adfa4a852e3adfd0620c33e78637b67ca1845a291f8b484cb769c76b67"
Jul 16 20:49:21.953221 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:21.953208    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673670832"
Jul 16 20:49:21.963000 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:49:21.962974    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7357-139849673670832\" is forbidden: User \"system:node:ip-10-0-128-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-109.ec2.internal' and this object" podUID="4a5e35c8-1a18-40be-b3ee-3d515a62c1d3" pod="opendatahub/test-curl-7357-139849673670832"
Jul 16 20:49:22.243431 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:49:22.243394    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7357-139849673670832\" is forbidden: User \"system:node:ip-10-0-128-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-109.ec2.internal' and this object" podUID="4a5e35c8-1a18-40be-b3ee-3d515a62c1d3" pod="opendatahub/test-curl-7357-139849673670832"
Jul 16 20:49:22.246735 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:22.246048    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a5e35c8-1a18-40be-b3ee-3d515a62c1d3" path="/var/lib/kubelet/pods/4a5e35c8-1a18-40be-b3ee-3d515a62c1d3/volumes"
Jul 16 20:49:22.959021 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:22.958983    2576 generic.go:356] "Generic (PLEG): container finished" podID="c28ff645-a7de-4ae8-9b6f-e36b3c758109" containerID="92673ee60da152121f5c436fe4ac5ee254253ed1b698e973216c20ad07aa5ead" exitCode=0
Jul 16 20:49:22.959204 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:22.959062    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7357-139849673757008" event={"ID":"c28ff645-a7de-4ae8-9b6f-e36b3c758109","Type":"ContainerDied","Data":"92673ee60da152121f5c436fe4ac5ee254253ed1b698e973216c20ad07aa5ead"}
Jul 16 20:49:24.103248 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.103225    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:24.123424 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.123399    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7357-139849673757008_c28ff645-a7de-4ae8-9b6f-e36b3c758109/test-curl-7357-139849673757008/0.log"
Jul 16 20:49:24.150168 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.150136    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7357-139849673757008"]
Jul 16 20:49:24.150418 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.150392    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p69b\" (UniqueName: \"kubernetes.io/projected/c28ff645-a7de-4ae8-9b6f-e36b3c758109-kube-api-access-5p69b\") pod \"c28ff645-a7de-4ae8-9b6f-e36b3c758109\" (UID: \"c28ff645-a7de-4ae8-9b6f-e36b3c758109\") "
Jul 16 20:49:24.152405 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.152379    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7357-139849673757008"]
Jul 16 20:49:24.152666 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.152644    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28ff645-a7de-4ae8-9b6f-e36b3c758109-kube-api-access-5p69b" (OuterVolumeSpecName: "kube-api-access-5p69b") pod "c28ff645-a7de-4ae8-9b6f-e36b3c758109" (UID: "c28ff645-a7de-4ae8-9b6f-e36b3c758109"). InnerVolumeSpecName "kube-api-access-5p69b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:49:24.245015 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.244927    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c28ff645-a7de-4ae8-9b6f-e36b3c758109" path="/var/lib/kubelet/pods/c28ff645-a7de-4ae8-9b6f-e36b3c758109/volumes"
Jul 16 20:49:24.251182 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.251149    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p69b\" (UniqueName: \"kubernetes.io/projected/c28ff645-a7de-4ae8-9b6f-e36b3c758109-kube-api-access-5p69b\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:49:24.302923 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.302894    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7357-139849673757008"]
Jul 16 20:49:24.303217 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.303205    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c28ff645-a7de-4ae8-9b6f-e36b3c758109" containerName="test-curl-7357-139849673757008"
Jul 16 20:49:24.303261 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.303219    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c28ff645-a7de-4ae8-9b6f-e36b3c758109" containerName="test-curl-7357-139849673757008"
Jul 16 20:49:24.303293 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.303285    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c28ff645-a7de-4ae8-9b6f-e36b3c758109" containerName="test-curl-7357-139849673757008"
Jul 16 20:49:24.306285 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.306267    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:24.312719 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.312673    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7357-139849673757008"]
Jul 16 20:49:24.352184 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.352155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jzpb\" (UniqueName: \"kubernetes.io/projected/c14165cf-0d8a-4935-8d7a-ae20e4445e7b-kube-api-access-9jzpb\") pod \"test-curl-7357-139849673757008\" (UID: \"c14165cf-0d8a-4935-8d7a-ae20e4445e7b\") " pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:24.452751 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.452694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jzpb\" (UniqueName: \"kubernetes.io/projected/c14165cf-0d8a-4935-8d7a-ae20e4445e7b-kube-api-access-9jzpb\") pod \"test-curl-7357-139849673757008\" (UID: \"c14165cf-0d8a-4935-8d7a-ae20e4445e7b\") " pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:24.461163 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.461138    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jzpb\" (UniqueName: \"kubernetes.io/projected/c14165cf-0d8a-4935-8d7a-ae20e4445e7b-kube-api-access-9jzpb\") pod \"test-curl-7357-139849673757008\" (UID: \"c14165cf-0d8a-4935-8d7a-ae20e4445e7b\") " pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:24.616314 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.616267    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:24.750722 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.750643    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7357-139849673757008"]
Jul 16 20:49:24.752912 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:49:24.752884    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc14165cf_0d8a_4935_8d7a_ae20e4445e7b.slice/crio-f7bd3ed7b8148876b4d48371f4b2530e31206f3be1ae97429dd2997732ff1a04 WatchSource:0}: Error finding container f7bd3ed7b8148876b4d48371f4b2530e31206f3be1ae97429dd2997732ff1a04: Status 404 returned error can't find the container with id f7bd3ed7b8148876b4d48371f4b2530e31206f3be1ae97429dd2997732ff1a04
Jul 16 20:49:24.970252 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.970168    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7357-139849673757008" event={"ID":"c14165cf-0d8a-4935-8d7a-ae20e4445e7b","Type":"ContainerStarted","Data":"f7bd3ed7b8148876b4d48371f4b2530e31206f3be1ae97429dd2997732ff1a04"}
Jul 16 20:49:24.971285 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.971264    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:24.971285 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:24.971284    2576 scope.go:117] "RemoveContainer" containerID="92673ee60da152121f5c436fe4ac5ee254253ed1b698e973216c20ad07aa5ead"
Jul 16 20:49:25.977435 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:25.977402    2576 generic.go:356] "Generic (PLEG): container finished" podID="c14165cf-0d8a-4935-8d7a-ae20e4445e7b" containerID="b8d2547f48ad814b497d89bd7d65e024a895ba8eb800d3fefc0e9d893e22ba4b" exitCode=0
Jul 16 20:49:25.977907 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:25.977455    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7357-139849673757008" event={"ID":"c14165cf-0d8a-4935-8d7a-ae20e4445e7b","Type":"ContainerDied","Data":"b8d2547f48ad814b497d89bd7d65e024a895ba8eb800d3fefc0e9d893e22ba4b"}
Jul 16 20:49:27.111248 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.111225    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:27.135898 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.135867    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7357-139849673757008_c14165cf-0d8a-4935-8d7a-ae20e4445e7b/test-curl-7357-139849673757008/0.log"
Jul 16 20:49:27.173756 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.173730    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jzpb\" (UniqueName: \"kubernetes.io/projected/c14165cf-0d8a-4935-8d7a-ae20e4445e7b-kube-api-access-9jzpb\") pod \"c14165cf-0d8a-4935-8d7a-ae20e4445e7b\" (UID: \"c14165cf-0d8a-4935-8d7a-ae20e4445e7b\") "
Jul 16 20:49:27.175913 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.175883    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14165cf-0d8a-4935-8d7a-ae20e4445e7b-kube-api-access-9jzpb" (OuterVolumeSpecName: "kube-api-access-9jzpb") pod "c14165cf-0d8a-4935-8d7a-ae20e4445e7b" (UID: "c14165cf-0d8a-4935-8d7a-ae20e4445e7b"). InnerVolumeSpecName "kube-api-access-9jzpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:49:27.185190 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.185156    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7357-139849673757008"]
Jul 16 20:49:27.190827 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.190809    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7357-139849673757008"]
Jul 16 20:49:27.274653 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.274576    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jzpb\" (UniqueName: \"kubernetes.io/projected/c14165cf-0d8a-4935-8d7a-ae20e4445e7b-kube-api-access-9jzpb\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:49:27.471182 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.471145    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7357-139849673757728"]
Jul 16 20:49:27.471491 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.471478    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c14165cf-0d8a-4935-8d7a-ae20e4445e7b" containerName="test-curl-7357-139849673757008"
Jul 16 20:49:27.471534 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.471494    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c14165cf-0d8a-4935-8d7a-ae20e4445e7b" containerName="test-curl-7357-139849673757008"
Jul 16 20:49:27.471586 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.471577    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c14165cf-0d8a-4935-8d7a-ae20e4445e7b" containerName="test-curl-7357-139849673757008"
Jul 16 20:49:27.474444 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.474428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:27.475947 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.475923    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpqp9\" (UniqueName: \"kubernetes.io/projected/4995ce96-23f8-4c6e-a336-9b7394ec08f7-kube-api-access-hpqp9\") pod \"test-curl-7357-139849673757728\" (UID: \"4995ce96-23f8-4c6e-a336-9b7394ec08f7\") " pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:27.480969 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.480950    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7357-139849673757728"]
Jul 16 20:49:27.576803 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.576776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpqp9\" (UniqueName: \"kubernetes.io/projected/4995ce96-23f8-4c6e-a336-9b7394ec08f7-kube-api-access-hpqp9\") pod \"test-curl-7357-139849673757728\" (UID: \"4995ce96-23f8-4c6e-a336-9b7394ec08f7\") " pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:27.585279 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.585256    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpqp9\" (UniqueName: \"kubernetes.io/projected/4995ce96-23f8-4c6e-a336-9b7394ec08f7-kube-api-access-hpqp9\") pod \"test-curl-7357-139849673757728\" (UID: \"4995ce96-23f8-4c6e-a336-9b7394ec08f7\") " pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:27.783980 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.783942    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:27.911894 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.911781    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7357-139849673757728"]
Jul 16 20:49:27.914309 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:49:27.914284    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4995ce96_23f8_4c6e_a336_9b7394ec08f7.slice/crio-24e4ccf5f25ad698259f217b934214c8af6c107c8911b93372fb89a26daa10f0 WatchSource:0}: Error finding container 24e4ccf5f25ad698259f217b934214c8af6c107c8911b93372fb89a26daa10f0: Status 404 returned error can't find the container with id 24e4ccf5f25ad698259f217b934214c8af6c107c8911b93372fb89a26daa10f0
Jul 16 20:49:27.987899 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.987873    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7bd3ed7b8148876b4d48371f4b2530e31206f3be1ae97429dd2997732ff1a04"
Jul 16 20:49:27.987899 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.987874    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:27.989101 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:27.989074    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7357-139849673757728" event={"ID":"4995ce96-23f8-4c6e-a336-9b7394ec08f7","Type":"ContainerStarted","Data":"24e4ccf5f25ad698259f217b934214c8af6c107c8911b93372fb89a26daa10f0"}
Jul 16 20:49:27.998245 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:49:27.998221    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7357-139849673757008\" is forbidden: User \"system:node:ip-10-0-128-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-109.ec2.internal' and this object" podUID="c14165cf-0d8a-4935-8d7a-ae20e4445e7b" pod="opendatahub/test-curl-7357-139849673757008"
Jul 16 20:49:28.244582 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.244559    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c14165cf-0d8a-4935-8d7a-ae20e4445e7b" path="/var/lib/kubelet/pods/c14165cf-0d8a-4935-8d7a-ae20e4445e7b/volumes"
Jul 16 20:49:28.496367 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.496278    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s" podUID="4c38418e-c685-468e-b313-c2f3b628be15" containerName="main" containerID="cri-o://9d646373b0d66f84d86c84661f3baef958f8bc27a8aee5bdc78f2db347ba4fba" gracePeriod=45
Jul 16 20:49:28.739077 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.739053    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:49:28.787126 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.787058    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-dshm\") pod \"4c38418e-c685-468e-b313-c2f3b628be15\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") "
Jul 16 20:49:28.787126 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.787092    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz9zb\" (UniqueName: \"kubernetes.io/projected/4c38418e-c685-468e-b313-c2f3b628be15-kube-api-access-nz9zb\") pod \"4c38418e-c685-468e-b313-c2f3b628be15\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") "
Jul 16 20:49:28.787126 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.787114    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-home\") pod \"4c38418e-c685-468e-b313-c2f3b628be15\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") "
Jul 16 20:49:28.787458 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.787161    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-model-cache\") pod \"4c38418e-c685-468e-b313-c2f3b628be15\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") "
Jul 16 20:49:28.787458 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.787205    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-tmp-dir\") pod \"4c38418e-c685-468e-b313-c2f3b628be15\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") "
Jul 16 20:49:28.787458 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.787230    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c38418e-c685-468e-b313-c2f3b628be15-tls-certs\") pod \"4c38418e-c685-468e-b313-c2f3b628be15\" (UID: \"4c38418e-c685-468e-b313-c2f3b628be15\") "
Jul 16 20:49:28.787590 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.787447    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-home" (OuterVolumeSpecName: "home") pod "4c38418e-c685-468e-b313-c2f3b628be15" (UID: "4c38418e-c685-468e-b313-c2f3b628be15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:49:28.787590 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.787479    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c38418e-c685-468e-b313-c2f3b628be15" (UID: "4c38418e-c685-468e-b313-c2f3b628be15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:49:28.787590 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.787517    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c38418e-c685-468e-b313-c2f3b628be15" (UID: "4c38418e-c685-468e-b313-c2f3b628be15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:49:28.789494 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.789473    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c38418e-c685-468e-b313-c2f3b628be15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c38418e-c685-468e-b313-c2f3b628be15" (UID: "4c38418e-c685-468e-b313-c2f3b628be15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:49:28.789771 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.789750    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-dshm" (OuterVolumeSpecName: "dshm") pod "4c38418e-c685-468e-b313-c2f3b628be15" (UID: "4c38418e-c685-468e-b313-c2f3b628be15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:49:28.789771 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.789757    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c38418e-c685-468e-b313-c2f3b628be15-kube-api-access-nz9zb" (OuterVolumeSpecName: "kube-api-access-nz9zb") pod "4c38418e-c685-468e-b313-c2f3b628be15" (UID: "4c38418e-c685-468e-b313-c2f3b628be15"). InnerVolumeSpecName "kube-api-access-nz9zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:49:28.888152 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.888123    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-home\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:49:28.888152 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.888150    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-model-cache\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:49:28.888152 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.888160    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-tmp-dir\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:49:28.888373 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.888168    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c38418e-c685-468e-b313-c2f3b628be15-tls-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:49:28.888373 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.888175    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c38418e-c685-468e-b313-c2f3b628be15-dshm\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:49:28.888373 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.888183    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz9zb\" (UniqueName: \"kubernetes.io/projected/4c38418e-c685-468e-b313-c2f3b628be15-kube-api-access-nz9zb\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:49:28.994882 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.994850    2576 generic.go:356] "Generic (PLEG): container finished" podID="4995ce96-23f8-4c6e-a336-9b7394ec08f7" containerID="1f8ad25021739b27117e46c18540743d32bddef207c2f03eef516e9c1cbce8a3" exitCode=0
Jul 16 20:49:28.995083 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.994937    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7357-139849673757728" event={"ID":"4995ce96-23f8-4c6e-a336-9b7394ec08f7","Type":"ContainerDied","Data":"1f8ad25021739b27117e46c18540743d32bddef207c2f03eef516e9c1cbce8a3"}
Jul 16 20:49:28.996236 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.996215    2576 generic.go:356] "Generic (PLEG): container finished" podID="4c38418e-c685-468e-b313-c2f3b628be15" containerID="9d646373b0d66f84d86c84661f3baef958f8bc27a8aee5bdc78f2db347ba4fba" exitCode=0
Jul 16 20:49:28.996326 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.996249    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"
Jul 16 20:49:28.996326 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.996250    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s" event={"ID":"4c38418e-c685-468e-b313-c2f3b628be15","Type":"ContainerDied","Data":"9d646373b0d66f84d86c84661f3baef958f8bc27a8aee5bdc78f2db347ba4fba"}
Jul 16 20:49:28.996326 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.996273    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s" event={"ID":"4c38418e-c685-468e-b313-c2f3b628be15","Type":"ContainerDied","Data":"786bfb8378601d409857b8a98a690aa8fb73138cc4521d457e388f9c9c9d31b0"}
Jul 16 20:49:28.996326 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:28.996289    2576 scope.go:117] "RemoveContainer" containerID="9d646373b0d66f84d86c84661f3baef958f8bc27a8aee5bdc78f2db347ba4fba"
Jul 16 20:49:29.005179 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:29.005162    2576 scope.go:117] "RemoveContainer" containerID="9d646373b0d66f84d86c84661f3baef958f8bc27a8aee5bdc78f2db347ba4fba"
Jul 16 20:49:29.005440 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:49:29.005420    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d646373b0d66f84d86c84661f3baef958f8bc27a8aee5bdc78f2db347ba4fba\": container with ID starting with 9d646373b0d66f84d86c84661f3baef958f8bc27a8aee5bdc78f2db347ba4fba not found: ID does not exist" containerID="9d646373b0d66f84d86c84661f3baef958f8bc27a8aee5bdc78f2db347ba4fba"
Jul 16 20:49:29.005511 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:29.005448    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d646373b0d66f84d86c84661f3baef958f8bc27a8aee5bdc78f2db347ba4fba"} err="failed to get container status \"9d646373b0d66f84d86c84661f3baef958f8bc27a8aee5bdc78f2db347ba4fba\": rpc error: code = NotFound desc = could not find container \"9d646373b0d66f84d86c84661f3baef958f8bc27a8aee5bdc78f2db347ba4fba\": container with ID starting with 9d646373b0d66f84d86c84661f3baef958f8bc27a8aee5bdc78f2db347ba4fba not found: ID does not exist"
Jul 16 20:49:29.031431 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:29.031400    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"]
Jul 16 20:49:29.038901 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:29.038850    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9b1256/rate-test-model-9b1256-kserve-7fb7695b5f-hrf5s"]
Jul 16 20:49:30.128460 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.128433    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:30.153596 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.153561    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7357-139849673757728_4995ce96-23f8-4c6e-a336-9b7394ec08f7/test-curl-7357-139849673757728/0.log"
Jul 16 20:49:30.186348 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.186313    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7357-139849673757728"]
Jul 16 20:49:30.189787 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.189763    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7357-139849673757728"]
Jul 16 20:49:30.197760 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.197737    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpqp9\" (UniqueName: \"kubernetes.io/projected/4995ce96-23f8-4c6e-a336-9b7394ec08f7-kube-api-access-hpqp9\") pod \"4995ce96-23f8-4c6e-a336-9b7394ec08f7\" (UID: \"4995ce96-23f8-4c6e-a336-9b7394ec08f7\") "
Jul 16 20:49:30.199910 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.199876    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4995ce96-23f8-4c6e-a336-9b7394ec08f7-kube-api-access-hpqp9" (OuterVolumeSpecName: "kube-api-access-hpqp9") pod "4995ce96-23f8-4c6e-a336-9b7394ec08f7" (UID: "4995ce96-23f8-4c6e-a336-9b7394ec08f7"). InnerVolumeSpecName "kube-api-access-hpqp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:49:30.244435 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.244403    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4995ce96-23f8-4c6e-a336-9b7394ec08f7" path="/var/lib/kubelet/pods/4995ce96-23f8-4c6e-a336-9b7394ec08f7/volumes"
Jul 16 20:49:30.244805 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.244791    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c38418e-c685-468e-b313-c2f3b628be15" path="/var/lib/kubelet/pods/4c38418e-c685-468e-b313-c2f3b628be15/volumes"
Jul 16 20:49:30.298847 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.298816    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpqp9\" (UniqueName: \"kubernetes.io/projected/4995ce96-23f8-4c6e-a336-9b7394ec08f7-kube-api-access-hpqp9\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:49:30.346499 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.346465    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7357-139849673757728"]
Jul 16 20:49:30.346846 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.346832    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4995ce96-23f8-4c6e-a336-9b7394ec08f7" containerName="test-curl-7357-139849673757728"
Jul 16 20:49:30.346888 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.346849    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4995ce96-23f8-4c6e-a336-9b7394ec08f7" containerName="test-curl-7357-139849673757728"
Jul 16 20:49:30.346888 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.346876    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c38418e-c685-468e-b313-c2f3b628be15" containerName="main"
Jul 16 20:49:30.346888 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.346882    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c38418e-c685-468e-b313-c2f3b628be15" containerName="main"
Jul 16 20:49:30.346974 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.346949    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4995ce96-23f8-4c6e-a336-9b7394ec08f7" containerName="test-curl-7357-139849673757728"
Jul 16 20:49:30.346974 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.346960    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c38418e-c685-468e-b313-c2f3b628be15" containerName="main"
Jul 16 20:49:30.349671 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.349657    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:30.356752 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.356576    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7357-139849673757728"]
Jul 16 20:49:30.399595 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.399563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49cb4\" (UniqueName: \"kubernetes.io/projected/b417a3e5-a508-4de2-a3d1-6803290e6ff7-kube-api-access-49cb4\") pod \"test-curl-7357-139849673757728\" (UID: \"b417a3e5-a508-4de2-a3d1-6803290e6ff7\") " pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:30.500166 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.500125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49cb4\" (UniqueName: \"kubernetes.io/projected/b417a3e5-a508-4de2-a3d1-6803290e6ff7-kube-api-access-49cb4\") pod \"test-curl-7357-139849673757728\" (UID: \"b417a3e5-a508-4de2-a3d1-6803290e6ff7\") " pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:30.508550 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.508486    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49cb4\" (UniqueName: \"kubernetes.io/projected/b417a3e5-a508-4de2-a3d1-6803290e6ff7-kube-api-access-49cb4\") pod \"test-curl-7357-139849673757728\" (UID: \"b417a3e5-a508-4de2-a3d1-6803290e6ff7\") " pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:30.665841 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.665803    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:30.803742 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:30.803673    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7357-139849673757728"]
Jul 16 20:49:30.805880 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:49:30.805851    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb417a3e5_a508_4de2_a3d1_6803290e6ff7.slice/crio-d716c84dad5e7a75471deb348d57be04b7e6bb35d685a5e7f62bc7b095ec44cc WatchSource:0}: Error finding container d716c84dad5e7a75471deb348d57be04b7e6bb35d685a5e7f62bc7b095ec44cc: Status 404 returned error can't find the container with id d716c84dad5e7a75471deb348d57be04b7e6bb35d685a5e7f62bc7b095ec44cc
Jul 16 20:49:31.007119 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:31.007090    2576 scope.go:117] "RemoveContainer" containerID="1f8ad25021739b27117e46c18540743d32bddef207c2f03eef516e9c1cbce8a3"
Jul 16 20:49:31.007286 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:31.007091    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:31.008456 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:31.008301    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7357-139849673757728" event={"ID":"b417a3e5-a508-4de2-a3d1-6803290e6ff7","Type":"ContainerStarted","Data":"d716c84dad5e7a75471deb348d57be04b7e6bb35d685a5e7f62bc7b095ec44cc"}
Jul 16 20:49:32.015063 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:32.015027    2576 generic.go:356] "Generic (PLEG): container finished" podID="b417a3e5-a508-4de2-a3d1-6803290e6ff7" containerID="03ceb49c7fbfbdc78462dd873205d2948cb20f4e41b37d58e211998ecc7dd009" exitCode=0
Jul 16 20:49:32.015474 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:32.015109    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7357-139849673757728" event={"ID":"b417a3e5-a508-4de2-a3d1-6803290e6ff7","Type":"ContainerDied","Data":"03ceb49c7fbfbdc78462dd873205d2948cb20f4e41b37d58e211998ecc7dd009"}
Jul 16 20:49:33.148100 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:33.148077    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:33.169409 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:33.169386    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7357-139849673757728_b417a3e5-a508-4de2-a3d1-6803290e6ff7/test-curl-7357-139849673757728/0.log"
Jul 16 20:49:33.190612 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:33.190582    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7357-139849673757728"]
Jul 16 20:49:33.196864 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:33.196841    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7357-139849673757728"]
Jul 16 20:49:33.224074 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:33.224048    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49cb4\" (UniqueName: \"kubernetes.io/projected/b417a3e5-a508-4de2-a3d1-6803290e6ff7-kube-api-access-49cb4\") pod \"b417a3e5-a508-4de2-a3d1-6803290e6ff7\" (UID: \"b417a3e5-a508-4de2-a3d1-6803290e6ff7\") "
Jul 16 20:49:33.226228 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:33.226206    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b417a3e5-a508-4de2-a3d1-6803290e6ff7-kube-api-access-49cb4" (OuterVolumeSpecName: "kube-api-access-49cb4") pod "b417a3e5-a508-4de2-a3d1-6803290e6ff7" (UID: "b417a3e5-a508-4de2-a3d1-6803290e6ff7"). InnerVolumeSpecName "kube-api-access-49cb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:49:33.325128 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:33.325077    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49cb4\" (UniqueName: \"kubernetes.io/projected/b417a3e5-a508-4de2-a3d1-6803290e6ff7-kube-api-access-49cb4\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:49:34.025567 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:34.025537    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:34.025781 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:34.025585    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d716c84dad5e7a75471deb348d57be04b7e6bb35d685a5e7f62bc7b095ec44cc"
Jul 16 20:49:34.035202 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:49:34.035175    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7357-139849673757728\" is forbidden: User \"system:node:ip-10-0-128-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-109.ec2.internal' and this object" podUID="b417a3e5-a508-4de2-a3d1-6803290e6ff7" pod="opendatahub/test-curl-7357-139849673757728"
Jul 16 20:49:34.244091 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:34.244054    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b417a3e5-a508-4de2-a3d1-6803290e6ff7" path="/var/lib/kubelet/pods/b417a3e5-a508-4de2-a3d1-6803290e6ff7/volumes"
Jul 16 20:49:59.252483 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:59.252452    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"]
Jul 16 20:49:59.253016 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:49:59.252678    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4" podUID="e0d58692-9661-4d7e-8b0b-b072136603a4" containerName="istio-proxy" containerID="cri-o://d65d3ea7759d659f031ceea867099018e468d0f0f5820f9ac44de97c22b41774" gracePeriod=30
Jul 16 20:50:04.498869 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.498844    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:50:04.598902 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.598876    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-podinfo\") pod \"e0d58692-9661-4d7e-8b0b-b072136603a4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") "
Jul 16 20:50:04.599071 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.598935    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-workload-socket\") pod \"e0d58692-9661-4d7e-8b0b-b072136603a4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") "
Jul 16 20:50:04.599071 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.598961    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v62zd\" (UniqueName: \"kubernetes.io/projected/e0d58692-9661-4d7e-8b0b-b072136603a4-kube-api-access-v62zd\") pod \"e0d58692-9661-4d7e-8b0b-b072136603a4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") "
Jul 16 20:50:04.599071 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.598980    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-token\") pod \"e0d58692-9661-4d7e-8b0b-b072136603a4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") "
Jul 16 20:50:04.599071 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.599030    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-workload-certs\") pod \"e0d58692-9661-4d7e-8b0b-b072136603a4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") "
Jul 16 20:50:04.599071 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.599045    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d58692-9661-4d7e-8b0b-b072136603a4-istiod-ca-cert\") pod \"e0d58692-9661-4d7e-8b0b-b072136603a4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") "
Jul 16 20:50:04.599071 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.599068    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-data\") pod \"e0d58692-9661-4d7e-8b0b-b072136603a4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") "
Jul 16 20:50:04.599392 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.599109    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-envoy\") pod \"e0d58692-9661-4d7e-8b0b-b072136603a4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") "
Jul 16 20:50:04.599392 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.599139    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-credential-socket\") pod \"e0d58692-9661-4d7e-8b0b-b072136603a4\" (UID: \"e0d58692-9661-4d7e-8b0b-b072136603a4\") "
Jul 16 20:50:04.599392 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.599298    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e0d58692-9661-4d7e-8b0b-b072136603a4" (UID: "e0d58692-9661-4d7e-8b0b-b072136603a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:50:04.599548 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.599466    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-workload-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:50:04.599548 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.599498    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e0d58692-9661-4d7e-8b0b-b072136603a4" (UID: "e0d58692-9661-4d7e-8b0b-b072136603a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:50:04.599679 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.599649    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e0d58692-9661-4d7e-8b0b-b072136603a4" (UID: "e0d58692-9661-4d7e-8b0b-b072136603a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:50:04.599770 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.599750    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d58692-9661-4d7e-8b0b-b072136603a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e0d58692-9661-4d7e-8b0b-b072136603a4" (UID: "e0d58692-9661-4d7e-8b0b-b072136603a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:50:04.600559 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.600526    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "e0d58692-9661-4d7e-8b0b-b072136603a4" (UID: "e0d58692-9661-4d7e-8b0b-b072136603a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:50:04.601436 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.601410    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e0d58692-9661-4d7e-8b0b-b072136603a4" (UID: "e0d58692-9661-4d7e-8b0b-b072136603a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 20:50:04.601521 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.601459    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "e0d58692-9661-4d7e-8b0b-b072136603a4" (UID: "e0d58692-9661-4d7e-8b0b-b072136603a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:50:04.601521 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.601466    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d58692-9661-4d7e-8b0b-b072136603a4-kube-api-access-v62zd" (OuterVolumeSpecName: "kube-api-access-v62zd") pod "e0d58692-9661-4d7e-8b0b-b072136603a4" (UID: "e0d58692-9661-4d7e-8b0b-b072136603a4"). InnerVolumeSpecName "kube-api-access-v62zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:50:04.601603 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.601545    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e0d58692-9661-4d7e-8b0b-b072136603a4" (UID: "e0d58692-9661-4d7e-8b0b-b072136603a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:50:04.700839 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.700810    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-credential-socket\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:50:04.700839 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.700835    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-podinfo\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:50:04.700839 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.700845    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v62zd\" (UniqueName: \"kubernetes.io/projected/e0d58692-9661-4d7e-8b0b-b072136603a4-kube-api-access-v62zd\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:50:04.701051 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.700854    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-token\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:50:04.701051 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.700864    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-workload-certs\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:50:04.701051 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.700873    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d58692-9661-4d7e-8b0b-b072136603a4-istiod-ca-cert\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:50:04.701051 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.700880    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-data\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:50:04.701051 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:04.700887    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d58692-9661-4d7e-8b0b-b072136603a4-istio-envoy\") on node \"ip-10-0-128-109.ec2.internal\" DevicePath \"\""
Jul 16 20:50:05.176688 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:05.176658    2576 generic.go:356] "Generic (PLEG): container finished" podID="e0d58692-9661-4d7e-8b0b-b072136603a4" containerID="d65d3ea7759d659f031ceea867099018e468d0f0f5820f9ac44de97c22b41774" exitCode=0
Jul 16 20:50:05.176906 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:05.176737    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"
Jul 16 20:50:05.176906 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:05.176736    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4" event={"ID":"e0d58692-9661-4d7e-8b0b-b072136603a4","Type":"ContainerDied","Data":"d65d3ea7759d659f031ceea867099018e468d0f0f5820f9ac44de97c22b41774"}
Jul 16 20:50:05.176906 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:05.176789    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4" event={"ID":"e0d58692-9661-4d7e-8b0b-b072136603a4","Type":"ContainerDied","Data":"d4f2a7937a7d428361266b7683ed6d357add3d433a45f029d8ee4def3f105b01"}
Jul 16 20:50:05.176906 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:05.176813    2576 scope.go:117] "RemoveContainer" containerID="d65d3ea7759d659f031ceea867099018e468d0f0f5820f9ac44de97c22b41774"
Jul 16 20:50:05.185745 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:05.185728    2576 scope.go:117] "RemoveContainer" containerID="d1875b744a22d6b520972f26337547639fd3c33bcb97b6aa0c5bf66cd9f2727d"
Jul 16 20:50:05.193083 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:05.193060    2576 scope.go:117] "RemoveContainer" containerID="d65d3ea7759d659f031ceea867099018e468d0f0f5820f9ac44de97c22b41774"
Jul 16 20:50:05.193304 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:50:05.193285    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d65d3ea7759d659f031ceea867099018e468d0f0f5820f9ac44de97c22b41774\": container with ID starting with d65d3ea7759d659f031ceea867099018e468d0f0f5820f9ac44de97c22b41774 not found: ID does not exist" containerID="d65d3ea7759d659f031ceea867099018e468d0f0f5820f9ac44de97c22b41774"
Jul 16 20:50:05.193347 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:05.193312    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d65d3ea7759d659f031ceea867099018e468d0f0f5820f9ac44de97c22b41774"} err="failed to get container status \"d65d3ea7759d659f031ceea867099018e468d0f0f5820f9ac44de97c22b41774\": rpc error: code = NotFound desc = could not find container \"d65d3ea7759d659f031ceea867099018e468d0f0f5820f9ac44de97c22b41774\": container with ID starting with d65d3ea7759d659f031ceea867099018e468d0f0f5820f9ac44de97c22b41774 not found: ID does not exist"
Jul 16 20:50:05.193347 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:05.193331    2576 scope.go:117] "RemoveContainer" containerID="d1875b744a22d6b520972f26337547639fd3c33bcb97b6aa0c5bf66cd9f2727d"
Jul 16 20:50:05.193559 ip-10-0-128-109 kubenswrapper[2576]: E0716 20:50:05.193542    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1875b744a22d6b520972f26337547639fd3c33bcb97b6aa0c5bf66cd9f2727d\": container with ID starting with d1875b744a22d6b520972f26337547639fd3c33bcb97b6aa0c5bf66cd9f2727d not found: ID does not exist" containerID="d1875b744a22d6b520972f26337547639fd3c33bcb97b6aa0c5bf66cd9f2727d"
Jul 16 20:50:05.193600 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:05.193565    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1875b744a22d6b520972f26337547639fd3c33bcb97b6aa0c5bf66cd9f2727d"} err="failed to get container status \"d1875b744a22d6b520972f26337547639fd3c33bcb97b6aa0c5bf66cd9f2727d\": rpc error: code = NotFound desc = could not find container \"d1875b744a22d6b520972f26337547639fd3c33bcb97b6aa0c5bf66cd9f2727d\": container with ID starting with d1875b744a22d6b520972f26337547639fd3c33bcb97b6aa0c5bf66cd9f2727d not found: ID does not exist"
Jul 16 20:50:05.200893 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:05.200869    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"]
Jul 16 20:50:05.207523 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:05.207504    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-9b1256-openshift-default-58d6bfdff9-jjpr4"]
Jul 16 20:50:06.244590 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:06.244560    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0d58692-9661-4d7e-8b0b-b072136603a4" path="/var/lib/kubelet/pods/e0d58692-9661-4d7e-8b0b-b072136603a4/volumes"
Jul 16 20:50:31.661093 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:31.661061    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rkllx_105589e3-1b76-4c01-af34-2757720c5814/manager/0.log"
Jul 16 20:50:32.411102 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:32.411075    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64c594d8d4-hxtq8_5d75dd15-733c-45c2-909b-11dcf95d5e07/copy-manifests/0.log"
Jul 16 20:50:32.430012 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:32.429979    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64c594d8d4-hxtq8_5d75dd15-733c-45c2-909b-11dcf95d5e07/manager/0.log"
Jul 16 20:50:32.917622 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:32.917590    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737215-7vxd2_eba5941e-f749-4b4b-acf5-e726ef6ee4d4/cleanup/0.log"
Jul 16 20:50:34.676170 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:34.676137    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-wklx4_184532ec-55fd-488e-8af4-38fc5a710ea5/postgres/0.log"
Jul 16 20:50:37.189400 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:37.189363    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf49qgp9_91d6c98c-0d9f-44e9-8654-d061efb6c892/istio-proxy/0.log"
Jul 16 20:50:37.591661 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:37.591631    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59f964d6d4-k6gpj_34b1e01f-1b19-428b-b13b-66d575c92f2b/kube-auth-proxy/0.log"
Jul 16 20:50:38.554167 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:38.554126    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-469cj_622e6119-a07a-41bc-8077-ab2e252667b6/main/0.log"
Jul 16 20:50:38.801153 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:38.801106    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db56wt7_34a007a9-185b-4fae-8b71-65475d381620/main/0.log"
Jul 16 20:50:38.919084 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:38.919059    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-lj67r_69a6aa6f-6049-4dfe-992f-1bec11157428/main/0.log"
Jul 16 20:50:44.828049 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.828016    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lw5rh/must-gather-kmwcz"]
Jul 16 20:50:44.828450 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.828327    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b417a3e5-a508-4de2-a3d1-6803290e6ff7" containerName="test-curl-7357-139849673757728"
Jul 16 20:50:44.828450 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.828339    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b417a3e5-a508-4de2-a3d1-6803290e6ff7" containerName="test-curl-7357-139849673757728"
Jul 16 20:50:44.828450 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.828353    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d58692-9661-4d7e-8b0b-b072136603a4" containerName="istio-proxy"
Jul 16 20:50:44.828450 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.828358    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d58692-9661-4d7e-8b0b-b072136603a4" containerName="istio-proxy"
Jul 16 20:50:44.828450 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.828366    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d58692-9661-4d7e-8b0b-b072136603a4" containerName="istio-proxy"
Jul 16 20:50:44.828450 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.828371    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d58692-9661-4d7e-8b0b-b072136603a4" containerName="istio-proxy"
Jul 16 20:50:44.828450 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.828429    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d58692-9661-4d7e-8b0b-b072136603a4" containerName="istio-proxy"
Jul 16 20:50:44.828450 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.828438    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d58692-9661-4d7e-8b0b-b072136603a4" containerName="istio-proxy"
Jul 16 20:50:44.828450 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.828446    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b417a3e5-a508-4de2-a3d1-6803290e6ff7" containerName="test-curl-7357-139849673757728"
Jul 16 20:50:44.835520 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.835495    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lw5rh/must-gather-kmwcz"
Jul 16 20:50:44.838452 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.838431    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lw5rh\"/\"kube-root-ca.crt\""
Jul 16 20:50:44.838929 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.838906    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lw5rh\"/\"openshift-service-ca.crt\""
Jul 16 20:50:44.839043 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.838906    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lw5rh\"/\"default-dockercfg-xznfh\""
Jul 16 20:50:44.844517 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.844495    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lw5rh/must-gather-kmwcz"]
Jul 16 20:50:44.949593 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.949564    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tk59\" (UniqueName: \"kubernetes.io/projected/52a2eaf9-402c-4819-9492-3ecdcccae65c-kube-api-access-8tk59\") pod \"must-gather-kmwcz\" (UID: \"52a2eaf9-402c-4819-9492-3ecdcccae65c\") " pod="openshift-must-gather-lw5rh/must-gather-kmwcz"
Jul 16 20:50:44.949767 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:44.949612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/52a2eaf9-402c-4819-9492-3ecdcccae65c-must-gather-output\") pod \"must-gather-kmwcz\" (UID: \"52a2eaf9-402c-4819-9492-3ecdcccae65c\") " pod="openshift-must-gather-lw5rh/must-gather-kmwcz"
Jul 16 20:50:45.050510 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:45.050475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tk59\" (UniqueName: \"kubernetes.io/projected/52a2eaf9-402c-4819-9492-3ecdcccae65c-kube-api-access-8tk59\") pod \"must-gather-kmwcz\" (UID: \"52a2eaf9-402c-4819-9492-3ecdcccae65c\") " pod="openshift-must-gather-lw5rh/must-gather-kmwcz"
Jul 16 20:50:45.050630 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:45.050540    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/52a2eaf9-402c-4819-9492-3ecdcccae65c-must-gather-output\") pod \"must-gather-kmwcz\" (UID: \"52a2eaf9-402c-4819-9492-3ecdcccae65c\") " pod="openshift-must-gather-lw5rh/must-gather-kmwcz"
Jul 16 20:50:45.050879 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:45.050862    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/52a2eaf9-402c-4819-9492-3ecdcccae65c-must-gather-output\") pod \"must-gather-kmwcz\" (UID: \"52a2eaf9-402c-4819-9492-3ecdcccae65c\") " pod="openshift-must-gather-lw5rh/must-gather-kmwcz"
Jul 16 20:50:45.059632 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:45.059604    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tk59\" (UniqueName: \"kubernetes.io/projected/52a2eaf9-402c-4819-9492-3ecdcccae65c-kube-api-access-8tk59\") pod \"must-gather-kmwcz\" (UID: \"52a2eaf9-402c-4819-9492-3ecdcccae65c\") " pod="openshift-must-gather-lw5rh/must-gather-kmwcz"
Jul 16 20:50:45.154204 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:45.154123    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lw5rh/must-gather-kmwcz"
Jul 16 20:50:45.291497 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:45.291464    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lw5rh/must-gather-kmwcz"]
Jul 16 20:50:45.294442 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:50:45.294414    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52a2eaf9_402c_4819_9492_3ecdcccae65c.slice/crio-f754f23c5d03e52ecdc977609b1ec9483ddbd6a0ee46fbd38f3202d89926f915 WatchSource:0}: Error finding container f754f23c5d03e52ecdc977609b1ec9483ddbd6a0ee46fbd38f3202d89926f915: Status 404 returned error can't find the container with id f754f23c5d03e52ecdc977609b1ec9483ddbd6a0ee46fbd38f3202d89926f915
Jul 16 20:50:45.346667 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:45.346635    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lw5rh/must-gather-kmwcz" event={"ID":"52a2eaf9-402c-4819-9492-3ecdcccae65c","Type":"ContainerStarted","Data":"f754f23c5d03e52ecdc977609b1ec9483ddbd6a0ee46fbd38f3202d89926f915"}
Jul 16 20:50:46.354452 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:46.354420    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lw5rh/must-gather-kmwcz" event={"ID":"52a2eaf9-402c-4819-9492-3ecdcccae65c","Type":"ContainerStarted","Data":"d69ed16658e6baab201652ab896b479a4b6f619469140b8eda511a2ca6ca793e"}
Jul 16 20:50:47.363214 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:47.363177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lw5rh/must-gather-kmwcz" event={"ID":"52a2eaf9-402c-4819-9492-3ecdcccae65c","Type":"ContainerStarted","Data":"9922bfd2d953f8fc49f41763a8507f82026a9d4ef16baaccf23a705347634c54"}
Jul 16 20:50:47.960452 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:47.960416    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-szlh8_f9711524-15cd-4165-854a-1453f37e4abc/global-pull-secret-syncer/0.log"
Jul 16 20:50:48.103945 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:48.103875    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m74x5_e4a3c35d-bfb2-4158-a8db-8b1d7f60c81c/konnectivity-agent/0.log"
Jul 16 20:50:48.134761 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:48.134728    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-109.ec2.internal_93b88c9554c3d1c5c57b49c2ee4f8f5e/haproxy/0.log"
Jul 16 20:50:56.071150 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:56.071110    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m62d4_f58e77e5-5e72-4ef6-a574-abfa4326dae4/node-exporter/0.log"
Jul 16 20:50:56.096448 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:56.096423    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m62d4_f58e77e5-5e72-4ef6-a574-abfa4326dae4/kube-rbac-proxy/0.log"
Jul 16 20:50:56.123573 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:56.123548    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m62d4_f58e77e5-5e72-4ef6-a574-abfa4326dae4/init-textfile/0.log"
Jul 16 20:50:56.317943 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:56.317915    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac34282-8050-4d86-8ab1-b2823b7fa1a5/prometheus/0.log"
Jul 16 20:50:56.359397 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:56.359316    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac34282-8050-4d86-8ab1-b2823b7fa1a5/config-reloader/0.log"
Jul 16 20:50:56.383321 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:56.383295    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac34282-8050-4d86-8ab1-b2823b7fa1a5/thanos-sidecar/0.log"
Jul 16 20:50:56.405171 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:56.405143    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac34282-8050-4d86-8ab1-b2823b7fa1a5/kube-rbac-proxy-web/0.log"
Jul 16 20:50:56.430773 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:56.430736    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac34282-8050-4d86-8ab1-b2823b7fa1a5/kube-rbac-proxy/0.log"
Jul 16 20:50:56.453974 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:56.453946    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac34282-8050-4d86-8ab1-b2823b7fa1a5/kube-rbac-proxy-thanos/0.log"
Jul 16 20:50:56.477802 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:56.477773    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac34282-8050-4d86-8ab1-b2823b7fa1a5/init-config-reloader/0.log"
Jul 16 20:50:56.508214 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:56.508181    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-78rw2_8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe/prometheus-operator/0.log"
Jul 16 20:50:56.531184 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:56.531136    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-78rw2_8ed5ac0b-45e3-433e-8d0d-a7809b55fbfe/kube-rbac-proxy/0.log"
Jul 16 20:50:56.559346 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:56.559317    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-z62x9_f4bad754-60b3-43ed-aae1-de6e5015c3ef/prometheus-operator-admission-webhook/0.log"
Jul 16 20:50:58.392756 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.392665    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lw5rh/must-gather-kmwcz" podStartSLOduration=13.428706872 podStartE2EDuration="14.39264088s" totalImagesPullingTime="963.934008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:50:47.386500992 +0000 UTC m=+3025.637121752" watchObservedRunningTime="2026-07-16 20:50:58.39264088 +0000 UTC m=+3036.643261631"
Jul 16 20:50:58.394287 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.394243    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"]
Jul 16 20:50:58.398622 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.398602    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.408610 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.408580    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"]
Jul 16 20:50:58.493364 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.493077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22725239-f8ad-409f-bd13-bfb54d545a21-sys\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.493364 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.493150    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22725239-f8ad-409f-bd13-bfb54d545a21-proc\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.493364 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.493178    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22725239-f8ad-409f-bd13-bfb54d545a21-lib-modules\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.493364 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.493224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hfnc\" (UniqueName: \"kubernetes.io/projected/22725239-f8ad-409f-bd13-bfb54d545a21-kube-api-access-9hfnc\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.493364 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.493251    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22725239-f8ad-409f-bd13-bfb54d545a21-podres\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.594087 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.594047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22725239-f8ad-409f-bd13-bfb54d545a21-proc\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.594087 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.594094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22725239-f8ad-409f-bd13-bfb54d545a21-lib-modules\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.594355 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.594137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hfnc\" (UniqueName: \"kubernetes.io/projected/22725239-f8ad-409f-bd13-bfb54d545a21-kube-api-access-9hfnc\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.594355 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.594166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22725239-f8ad-409f-bd13-bfb54d545a21-podres\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.594355 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.594254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22725239-f8ad-409f-bd13-bfb54d545a21-sys\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.594513 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.594357    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22725239-f8ad-409f-bd13-bfb54d545a21-sys\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.594513 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.594415    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22725239-f8ad-409f-bd13-bfb54d545a21-proc\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.594622 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.594531    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22725239-f8ad-409f-bd13-bfb54d545a21-lib-modules\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.594829 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.594805    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22725239-f8ad-409f-bd13-bfb54d545a21-podres\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.605387 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.605345    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hfnc\" (UniqueName: \"kubernetes.io/projected/22725239-f8ad-409f-bd13-bfb54d545a21-kube-api-access-9hfnc\") pod \"perf-node-gather-daemonset-bqf2m\" (UID: \"22725239-f8ad-409f-bd13-bfb54d545a21\") " pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.713789 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.713676    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:58.887690 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:58.887664    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"]
Jul 16 20:50:58.890467 ip-10-0-128-109 kubenswrapper[2576]: W0716 20:50:58.890438    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod22725239_f8ad_409f_bd13_bfb54d545a21.slice/crio-45f4fc7438c05b2259dfe40a419c5732e87c27a0e0dd47ba85cb4c0607dfda47 WatchSource:0}: Error finding container 45f4fc7438c05b2259dfe40a419c5732e87c27a0e0dd47ba85cb4c0607dfda47: Status 404 returned error can't find the container with id 45f4fc7438c05b2259dfe40a419c5732e87c27a0e0dd47ba85cb4c0607dfda47
Jul 16 20:50:59.234366 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:59.234274    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-m8ht5_10f346ba-a76a-45c8-bab6-3591fa1dcccd/download-server/0.log"
Jul 16 20:50:59.453359 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:59.453322    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m" event={"ID":"22725239-f8ad-409f-bd13-bfb54d545a21","Type":"ContainerStarted","Data":"67fd7dcafe2d4afaaeea45c86d1f403deccd5982be303efd2b9a848b9ad9d455"}
Jul 16 20:50:59.453861 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:59.453379    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m" event={"ID":"22725239-f8ad-409f-bd13-bfb54d545a21","Type":"ContainerStarted","Data":"45f4fc7438c05b2259dfe40a419c5732e87c27a0e0dd47ba85cb4c0607dfda47"}
Jul 16 20:50:59.453861 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:59.453447    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:50:59.481071 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:50:59.481020    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m" podStartSLOduration=1.481003597 podStartE2EDuration="1.481003597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:50:59.478813153 +0000 UTC m=+3037.729433911" watchObservedRunningTime="2026-07-16 20:50:59.481003597 +0000 UTC m=+3037.731624355"
Jul 16 20:51:01.001692 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:01.001663    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l4b9r_f59179dd-36c5-404f-837f-8e54b1c799ed/dns/0.log"
Jul 16 20:51:01.030720 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:01.030667    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l4b9r_f59179dd-36c5-404f-837f-8e54b1c799ed/kube-rbac-proxy/0.log"
Jul 16 20:51:01.164542 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:01.164515    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vqw9q_08c98ae9-ea2f-4de5-9b8b-06cc63a945d7/dns-node-resolver/0.log"
Jul 16 20:51:01.723445 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:01.723409    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f46679889-f9szw_b2475315-9baa-456c-b86a-c97a0f1a04a4/registry/0.log"
Jul 16 20:51:01.748827 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:01.748798    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5h765_97cc28a3-e813-4407-bad7-8f53dec993f4/node-ca/0.log"
Jul 16 20:51:02.864778 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:02.864748    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf49qgp9_91d6c98c-0d9f-44e9-8654-d061efb6c892/istio-proxy/0.log"
Jul 16 20:51:03.134332 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:03.134242    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59f964d6d4-k6gpj_34b1e01f-1b19-428b-b13b-66d575c92f2b/kube-auth-proxy/0.log"
Jul 16 20:51:04.102679 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:04.102645    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l4947_bb00e945-1b1b-4387-8834-cd0365bdcd23/serve-healthcheck-canary/0.log"
Jul 16 20:51:04.791911 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:04.791864    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4lrz_13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1/kube-rbac-proxy/0.log"
Jul 16 20:51:04.823586 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:04.823517    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4lrz_13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1/exporter/0.log"
Jul 16 20:51:04.847725 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:04.847674    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4lrz_13ef9ffe-f7cd-4baf-826b-c05b6b5f8ea1/extractor/0.log"
Jul 16 20:51:05.471368 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:05.471338    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lw5rh/perf-node-gather-daemonset-bqf2m"
Jul 16 20:51:07.359097 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:07.359056    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rkllx_105589e3-1b76-4c01-af34-2757720c5814/manager/0.log"
Jul 16 20:51:07.890431 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:07.890401    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64c594d8d4-hxtq8_5d75dd15-733c-45c2-909b-11dcf95d5e07/manager/0.log"
Jul 16 20:51:07.912854 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:07.912826    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64c594d8d4-hxtq8_5d75dd15-733c-45c2-909b-11dcf95d5e07/copy-manifests/0.log"
Jul 16 20:51:08.384531 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:08.384499    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf49qgp9_91d6c98c-0d9f-44e9-8654-d061efb6c892/istio-proxy/0.log"
Jul 16 20:51:08.630231 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:08.630192    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59f964d6d4-k6gpj_34b1e01f-1b19-428b-b13b-66d575c92f2b/kube-auth-proxy/0.log"
Jul 16 20:51:09.194234 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:09.194206    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7b8db6765-p48sw_43802dab-ca6b-4172-ad93-cbbeff6cff69/manager/0.log"
Jul 16 20:51:10.020656 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:10.020581    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf49qgp9_91d6c98c-0d9f-44e9-8654-d061efb6c892/istio-proxy/0.log"
Jul 16 20:51:10.263257 ip-10-0-128-109 kubenswrapper[2576]: I0716 20:51:10.263226    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59f964d6d4-k6gpj_34b1e01f-1b19-428b-b13b-66d575c92f2b/kube-auth-proxy/0.log"