Jul 29 20:44:25.944140 ip-10-0-129-194 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 29 20:44:25.944154 ip-10-0-129-194 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 29 20:44:25.944165 ip-10-0-129-194 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 29 20:44:25.944514 ip-10-0-129-194 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 29 20:44:35.197680 ip-10-0-129-194 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Jul 29 20:44:35.197695 ip-10-0-129-194 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6c9b6ddc528f4f60887afe90827d5db8 --
Jul 29 20:46:23.247121 ip-10-0-129-194 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 20:46:23.579859 ip-10-0-129-194 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 20:46:23.579859 ip-10-0-129-194 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 20:46:23.579859 ip-10-0-129-194 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 20:46:23.579859 ip-10-0-129-194 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 20:46:23.579859 ip-10-0-129-194 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 20:46:23.580809 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.580706    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 20:46:23.584690 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.584670    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:46:23.584690 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.584689    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:46:23.584761 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.584693    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:46:23.584761 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.584696    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:46:23.584761 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.584702    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:46:23.584761 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.584707    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:46:23.584761 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.584711    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:46:23.584761 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.584714    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:46:23.584761 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.584717    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:46:23.584761 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.584720    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:46:23.584761 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.584723    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:46:23.586017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586007    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586021    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586025    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586028    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586031    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586034    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586037    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586040    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586043    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586046    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586049    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586052    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586055    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586058    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:46:23.586055 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586060    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586064    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586067    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586069    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586072    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586074    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586077    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586079    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586082    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586084    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586087    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586089    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586092    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586094    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586096    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586099    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586102    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586104    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586107    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586110    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:46:23.586423 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586112    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586114    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586117    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586120    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586122    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586126    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586128    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586131    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586133    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586136    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586139    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586143    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586147    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586150    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586153    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586156    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586159    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586162    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586165    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:46:23.586945 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586168    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586171    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586175    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586178    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586180    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586183    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586186    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586189    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586192    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586194    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586197    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586199    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586202    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586206    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586209    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586212    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586215    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586217    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586220    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586222    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:46:23.587412 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586225    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586227    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586230    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586233    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586235    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586239    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586241    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586244    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586247    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586249    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586252    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586254    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586257    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586260    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586263    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.586997    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587004    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587007    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587010    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587013    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:46:23.587920 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587017    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587020    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587025    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587028    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587032    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587034    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587037    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587040    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587042    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587045    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587048    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587050    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587053    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587056    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587058    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587061    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587064    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587068    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587071    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:46:23.588411 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587076    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587080    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587083    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587086    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587089    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587092    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587095    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587098    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587101    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587104    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587107    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587110    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587113    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587117    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587121    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587124    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587127    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587129    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587132    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587135    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:46:23.588970 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587138    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587141    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587143    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587146    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587148    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587151    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587153    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587156    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587161    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587164    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587167    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587170    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587173    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587175    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587178    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587181    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587184    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587186    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587189    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587191    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:46:23.589502 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587194    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587197    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587199    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587202    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587204    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587207    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587209    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587213    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587215    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587218    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587221    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587223    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587226    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587228    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587231    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587234    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587237    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587240    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587242    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587244    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:46:23.590014 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587248    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587253    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587257    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587259    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587262    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587265    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587267    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587270    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587273    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587275    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587278    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587281    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587283    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587286    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587288    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587694    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587700    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587703    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587705    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587709    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:46:23.590505 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587711    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587714    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587717    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587719    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587722    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587725    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587727    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587730    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587733    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587735    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587738    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587740    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587745    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587748    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587751    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587753    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587756    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587759    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587761    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587764    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:46:23.591030 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587779    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587783    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587787    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587791    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587794    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587797    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587800    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587803    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587806    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587809    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587811    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587814    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587818    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587820    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587823    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587825    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587828    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587830    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587833    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:46:23.591535 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587836    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587838    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587841    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587843    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587846    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587850    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587853    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587856    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587859    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587861    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587865    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587869    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587872    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587875    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587878    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587881    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587884    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587887    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587889    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:46:23.592017 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587893    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587896    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587899    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587902    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587905    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587907    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587910    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587914    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587916    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587919    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587922    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587924    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587927    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587930    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587932    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587935    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587938    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587940    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587944    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587947    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:46:23.592498 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587950    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587954    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587957    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587959    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587962    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587965    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587968    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587970    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587973    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587975    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587978    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587980    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587983    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587985    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587987    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.587990    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588072    2584 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588083    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588092    2584 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588101    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588110    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588116    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 20:46:23.593007 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588122    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588128    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588133    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588138    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588143    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588146    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588150    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588153    2584 flags.go:64] FLAG: --cgroup-root=""
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588156    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588164    2584 flags.go:64] FLAG: --client-ca-file=""
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588167    2584 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588170    2584 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588176    2584 flags.go:64] FLAG: --cluster-domain=""
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588179    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588182    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588185    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588189    2584 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588193    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588196    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588199    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588202    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588206    2584 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588209    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588212    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 20:46:23.593546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588215    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588218    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588222    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588225    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588229    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588231    2584 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588235    2584 flags.go:64] FLAG: --enable-server="true"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588244    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588249    2584 flags.go:64] FLAG: --event-burst="100"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588252    2584 flags.go:64] FLAG: --event-qps="50"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588254    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588257    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588261    2584 flags.go:64] FLAG: --eviction-hard=""
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588265    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588268    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588271    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588274    2584 flags.go:64] FLAG: --eviction-soft=""
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588278    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588281    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588284    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588287    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588290    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588293    2584 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588296    2584 flags.go:64] FLAG: --feature-gates=""
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588300    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 20:46:23.594140 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588303    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588306    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588309    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588312    2584 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588315    2584 flags.go:64] FLAG: --help="false"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588318    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-129-194.ec2.internal"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588321    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588324    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588327    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588331    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588334    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588337    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588340    2584 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588343    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588346    2584 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588354    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588358    2584 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588361    2584 flags.go:64] FLAG: --kube-reserved=""
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588364    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588367    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588370    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588373    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588375    2584 flags.go:64] FLAG: --lock-file=""
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588378    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 20:46:23.594865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588383    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588386    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588391    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588394    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588397    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588400    2584 flags.go:64] FLAG: --logging-format="text"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588403    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588406    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588409    2584 flags.go:64] FLAG: --manifest-url=""
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588412    2584 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588416    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588420    2584 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588424    2584 flags.go:64] FLAG: --max-pods="110"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588427    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588430    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588433    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588436    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588439    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588442    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588445    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588453    2584 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588456    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588459    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 20:46:23.595467 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588462    2584 flags.go:64] FLAG: --pod-cidr=""
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588472    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588478    2584 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588481    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588484    2584 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588488    2584 flags.go:64] FLAG: --port="10250"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588491    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588494    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0121b1546f9d216a6"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588498    2584 flags.go:64] FLAG: --qos-reserved=""
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588503    2584 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588506    2584 flags.go:64] FLAG: --register-node="true"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588509    2584 flags.go:64] FLAG: --register-with-taints=""
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588512    2584 flags.go:64] FLAG: --registry-burst="10"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588515    2584 flags.go:64] FLAG: --registry-qps="5"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588518    2584 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588521    2584 flags.go:64] FLAG: --reserved-memory=""
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588525    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588528    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588531    2584 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588534    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588537    2584 flags.go:64] FLAG: --runonce="false"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588540    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588543    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588546    2584 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588549    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 20:46:23.596072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588552    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588555    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588558    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588562    2584 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588564    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588567    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588570    2584 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588573    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588576    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588582    2584 flags.go:64] FLAG: --system-cgroups=""
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588585    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588591    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588595    2584 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588599    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588626    2584 flags.go:64] FLAG: --tls-min-version=""
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588631    2584 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588636    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588639    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588642    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588645    2584 flags.go:64] FLAG: --v="2"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588650    2584 flags.go:64] FLAG: --version="false"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588653    2584 flags.go:64] FLAG: --vmodule=""
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588657    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.588661    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588833    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:46:23.596688 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588838    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588841    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588845    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588848    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588851    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588853    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588856    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588859    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588861    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588864    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588866    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588869    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588872    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588875    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588890    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588893    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588897    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588900    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588902    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588907    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:46:23.597341 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588910    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588914    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588917    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588920    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588923    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588926    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588929    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588931    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588934    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588937    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588939    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588942    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588945    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588947    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588950    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588953    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588955    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588958    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588961    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588963    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:46:23.597860 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588966    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588970    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588973    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588978    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588981    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588984    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588987    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588990    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588993    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588995    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.588998    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589002    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589005    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589007    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589010    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589013    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589016    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589019    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589021    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:46:23.598401 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589024    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589026    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589029    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589031    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589034    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589037    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589040    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589042    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589045    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589048    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589050    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589053    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589055    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589058    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589061    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589063    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589065    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589068    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589070    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:46:23.598898 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589073    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589075    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589079    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589082    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589085    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589090    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589092    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589095    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589097    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589101    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589104    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589107    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589110    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589112    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589115    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589118    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589120    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589123    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589126    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:46:23.599372 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.589128    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.589925    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.596391    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.596408    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.596430    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.596436    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596506    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596512    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596515    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596518    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596522    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596525    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596528    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596531    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596534    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:46:23.599875 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596537    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596539    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596542    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596544    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596547    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596550    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596553    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596555    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596558    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596560    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596563    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596565    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596568    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596570    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596573    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596576    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596578    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596582    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596586    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596589    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:46:23.600263 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596593    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596596    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596599    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596601    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596604    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596607    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596609    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596612    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596614    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596617    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596620    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596622    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596625    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596628    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596630    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596633    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596635    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596638    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596640    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:46:23.600790 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596643    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596645    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596648    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596651    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596653    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596656    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596658    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596661    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596664    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596667    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596670    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596673    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596676    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596679    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596683    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596686    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596688    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596691    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596694    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:46:23.601271 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596697    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596699    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596702    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596707    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596711    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596714    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596717    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596719    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596723    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596725    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596728    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596730    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596733    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596735    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596738    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596741    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596743    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596746    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596748    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596751    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:46:23.601801 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596753    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596757    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596761    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596764    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596784    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596790    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596794    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596799    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596804    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596806    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596810    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596812    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.596818    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596954    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596960    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:46:23.602314 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596966    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596969    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596972    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596975    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596978    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596980    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596984    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596987    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596990    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596993    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596997    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.596999    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597002    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597005    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597008    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597010    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597013    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597015    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597018    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597020    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:46:23.602695 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597023    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597026    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597028    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597031    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597034    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597037    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597040    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597042    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597046    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597048    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597051    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597054    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597056    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597059    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597061    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597063    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597066    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597068    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597071    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:46:23.603532 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597073    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597076    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597078    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597081    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597083    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597085    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597088    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597090    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597093    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597095    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597098    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597101    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597103    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597106    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597108    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597111    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597113    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597116    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597119    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597122    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:46:23.604117 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597125    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597127    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597129    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597132    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597134    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597137    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597139    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597142    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597144    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597146    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597149    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597151    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597155    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597158    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597161    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597163    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597166    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597168    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597171    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:46:23.604611 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597173    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597176    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597178    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597181    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597184    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597186    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597189    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597191    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597193    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597196    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597199    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597202    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597204    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597207    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597210    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597212    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597215    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597217    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:46:23.605101 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:23.597220    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:46:23.605553 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.597224    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 20:46:23.605553 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.597348    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 20:46:23.605553 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.600440    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 20:46:23.605553 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.601575    2584 server.go:1019] "Starting client certificate rotation"
Jul 29 20:46:23.605553 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.602322    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 20:46:23.605553 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.602374    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 20:46:23.606103 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.606075    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 20:46:23.611052 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.611032    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 20:46:23.623931 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.623912    2584 log.go:25] "Validated CRI v1 runtime API"
Jul 29 20:46:23.626181 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.626165    2584 log.go:25] "Validated CRI v1 image API"
Jul 29 20:46:23.626942 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.626924    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 20:46:23.629052 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.629029    2584 fs.go:135] Filesystem UUIDs: map[000f31f4-6cbf-4ee1-83c2-8ed54b8fd57a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fca78bb0-d292-4476-a8bd-d009bc7383e7:/dev/nvme0n1p4]
Jul 29 20:46:23.629133 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.629050    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 20:46:23.630701 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.630684    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 20:46:23.633950 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.633842    2584 manager.go:216] Machine: {Timestamp:2026-07-29 20:46:23.632631338 +0000 UTC m=+0.283297521 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102141 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21e859c38e84890778f61cd9fef7e9 SystemUUID:ec21e859-c38e-8489-0778-f61cd9fef7e9 BootID:6c9b6ddc-528f-4f60-887a-fe90827d5db8 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:19:47:da:c8:f7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:19:47:da:c8:f7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:f6:ec:cb:7c:0f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 20:46:23.633950 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.633944    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 20:46:23.634078 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.634064    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 20:46:23.636475 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.636446    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 20:46:23.636671 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.636477    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 20:46:23.637170 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.637158    2584 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 20:46:23.637234 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.637177    2584 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 20:46:23.637234 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.637187    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 20:46:23.637803 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.637788    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 20:46:23.638608 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.638596    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 20:46:23.641260 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.641245    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 20:46:23.641401 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.641390    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 20:46:23.641461 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.641428    2584 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 20:46:23.641461 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.641443    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 20:46:23.641461 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.641458    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 20:46:23.641577 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.641474    2584 kubelet.go:398] "Adding apiserver pod source"
Jul 29 20:46:23.641577 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.641489    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 20:46:23.641882 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.641869    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 20:46:23.641953 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.641892    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 20:46:23.643887 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.643873    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 20:46:23.646236 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.646215    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 20:46:23.646342 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.646251    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 20:46:23.647765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.647734    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 20:46:23.647765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.647753    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 20:46:23.647765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.647760    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 20:46:23.647765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.647779    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 20:46:23.647924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.647790    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 20:46:23.647924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.647796    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 20:46:23.647924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.647804    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 20:46:23.647924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.647809    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 20:46:23.647924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.647816    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 20:46:23.647924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.647822    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 20:46:23.647924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.647830    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 20:46:23.647924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.647836    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 20:46:23.648486 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.648476    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 20:46:23.648486 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.648484    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 20:46:23.650852 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.650832    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 20:46:23.651405 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:23.651385    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 20:46:23.651457 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:23.651389    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 20:46:23.651614 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.651596    2584 server.go:1268] "Started kubelet"
Jul 29 20:46:23.651738 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.651678    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 20:46:23.651738 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.651684    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 20:46:23.651870 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.651759    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 20:46:23.652566 ip-10-0-129-194 systemd[1]: Started Kubernetes Kubelet.
Jul 29 20:46:23.652939 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.652923    2584 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 20:46:23.653345 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.653331    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 20:46:23.655731 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.655710    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w8cct"
Jul 29 20:46:23.656885 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:23.656859    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 20:46:23.657327 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:23.656569    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-194.ec2.internal.18c6dda07c75a0b5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-194.ec2.internal,UID:ip-10-0-129-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-194.ec2.internal,},FirstTimestamp:2026-07-29 20:46:23.651569845 +0000 UTC m=+0.302236008,LastTimestamp:2026-07-29 20:46:23.651569845 +0000 UTC m=+0.302236008,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-194.ec2.internal,}"
Jul 29 20:46:23.659432 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.659418    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 20:46:23.659482 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.659428    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 20:46:23.659527 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:23.659496    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Jul 29 20:46:23.659980 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.659963    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 20:46:23.659980 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.659968    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 20:46:23.660138 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.659999    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 20:46:23.660291 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.660275    2584 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 20:46:23.660291 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.660289    2584 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 20:46:23.661335 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.661318    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w8cct"
Jul 29 20:46:23.662227 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.661968    2584 factory.go:55] Registering systemd factory
Jul 29 20:46:23.662227 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.661985    2584 factory.go:223] Registration of the systemd container factory successfully
Jul 29 20:46:23.662438 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.662261    2584 factory.go:153] Registering CRI-O factory
Jul 29 20:46:23.662438 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.662273    2584 factory.go:223] Registration of the crio container factory successfully
Jul 29 20:46:23.662438 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.662313    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 20:46:23.662438 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.662335    2584 factory.go:103] Registering Raw factory
Jul 29 20:46:23.662438 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.662349    2584 manager.go:1196] Started watching for new ooms in manager
Jul 29 20:46:23.662925 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.662908    2584 manager.go:349] Starting recovery of all containers
Jul 29 20:46:23.666471 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:23.666442    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 20:46:23.666471 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:23.666426    2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 20:46:23.675244 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.675226    2584 manager.go:354] Recovery completed
Jul 29 20:46:23.679249 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.679236    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:46:23.681456 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.681440    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:46:23.681519 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.681471    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:46:23.681519 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.681481    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:46:23.682237 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.682215    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 20:46:23.682237 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.682231    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 20:46:23.682340 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.682221    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:46:23.682340 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.682252    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 20:46:23.683193 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.683175    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:46:23.683254 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.683211    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:46:23.683254 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.683227    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:46:23.684742 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.684730    2584 policy_none.go:49] "None policy: Start"
Jul 29 20:46:23.684799 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.684747    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 20:46:23.684799 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.684758    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 20:46:23.686687 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.686672    2584 policy_none.go:47] "Start"
Jul 29 20:46:23.721525 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.721338    2584 manager.go:337] "Starting Device Plugin manager"
Jul 29 20:46:23.721645 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:23.721554    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 20:46:23.721645 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.721567    2584 server.go:87] "Starting device plugin registration server"
Jul 29 20:46:23.721864 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.721853    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 20:46:23.721905 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.721867    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 20:46:23.721983 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.721963    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 20:46:23.722070 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.722053    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 20:46:23.722070 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.722066    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 20:46:23.722556 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:23.722534    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 20:46:23.722640 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:23.722583    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-194.ec2.internal\" not found"
Jul 29 20:46:23.803330 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.803281    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 20:46:23.805496 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.804569    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 20:46:23.805496 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.804606    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 20:46:23.805496 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.804631    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 20:46:23.805496 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:23.804666    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 20:46:23.807168 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.807149    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:46:23.822541 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.822521    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:46:23.823457 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.823433    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:46:23.823581 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.823467    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:46:23.823581 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.823482    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:46:23.823581 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.823508    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-194.ec2.internal"
Jul 29 20:46:23.830935 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:23.830875    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-194.ec2.internal"
Jul 29 20:46:23.830935 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:23.830901    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-194.ec2.internal\": node \"ip-10-0-129-194.ec2.internal\" not found"
Jul 29 20:46:23.841727 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:23.841697    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Jul 29 20:46:24.959792 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.959759    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"]
Jul 29 20:46:24.959888 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.959847    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:46:24.961276 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.961261    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:46:24.961333 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.961290    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:46:24.961333 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.961300    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:46:24.963537 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.963526    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:46:24.963639 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.963626    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:24.963680 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.963654    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:46:24.964233 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.964207    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:46:24.964233 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.964235    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:46:24.964350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.964244    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:46:24.964695 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.964681    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:46:24.964754 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.964704    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:46:24.964754 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.964714    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:46:24.966281 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.966265    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:24.966360 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.966295    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:46:24.966940 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.966926    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:46:24.967008 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.966951    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:46:24.967008 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:24.966963    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:46:24.982753 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:24.982724    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-194.ec2.internal\" not found" node="ip-10-0-129-194.ec2.internal"
Jul 29 20:46:24.986829 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:24.986810    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-194.ec2.internal\" not found" node="ip-10-0-129-194.ec2.internal"
Jul 29 20:46:24.996544 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:24.996528    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Jul 29 20:46:25.016380 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.016351    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e50be1114c5d989169bea87a3d8b446-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal\" (UID: \"9e50be1114c5d989169bea87a3d8b446\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:25.016380 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.016382    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e50be1114c5d989169bea87a3d8b446-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal\" (UID: \"9e50be1114c5d989169bea87a3d8b446\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:25.016549 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.016398    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d5318fce92e6dc30124e077fbe98465-config\") pod \"kube-apiserver-proxy-ip-10-0-129-194.ec2.internal\" (UID: \"5d5318fce92e6dc30124e077fbe98465\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:25.096982 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.096947    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Jul 29 20:46:25.117431 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.117403    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e50be1114c5d989169bea87a3d8b446-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal\" (UID: \"9e50be1114c5d989169bea87a3d8b446\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:25.117431 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.117422    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e50be1114c5d989169bea87a3d8b446-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal\" (UID: \"9e50be1114c5d989169bea87a3d8b446\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:25.117542 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.117454    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e50be1114c5d989169bea87a3d8b446-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal\" (UID: \"9e50be1114c5d989169bea87a3d8b446\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:25.117542 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.117481    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d5318fce92e6dc30124e077fbe98465-config\") pod \"kube-apiserver-proxy-ip-10-0-129-194.ec2.internal\" (UID: \"5d5318fce92e6dc30124e077fbe98465\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:25.117542 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.117502    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e50be1114c5d989169bea87a3d8b446-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal\" (UID: \"9e50be1114c5d989169bea87a3d8b446\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:25.117542 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.117511    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d5318fce92e6dc30124e077fbe98465-config\") pod \"kube-apiserver-proxy-ip-10-0-129-194.ec2.internal\" (UID: \"5d5318fce92e6dc30124e077fbe98465\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:25.197773 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.197706    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Jul 29 20:46:25.285253 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.285210    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:25.290048 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.289922    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:25.298760 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.298739    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Jul 29 20:46:25.399299 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.399257    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Jul 29 20:46:25.499837 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.499756    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-194.ec2.internal\" not found"
Jul 29 20:46:25.560781 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.560752    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:46:25.614863 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.614830    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:25.625615 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.625590    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 20:46:25.627421 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.627406    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal"
Jul 29 20:46:25.637274 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.637252    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 20:46:25.655434 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.655403    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 20:46:25.655607 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.655578    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 20:46:25.655664 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.655589    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 20:46:25.690575 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.690548    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:46:25.694060 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.694027    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:46:25.696393 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.696375    2584 apiserver.go:52] "Watching apiserver"
Jul 29 20:46:25.700015 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.699998    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 20:46:25.700368 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.700345    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-m4fbv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm","openshift-cluster-node-tuning-operator/tuned-wvq8v","openshift-dns/node-resolver-g2z8d","openshift-image-registry/node-ca-562n4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal","openshift-network-operator/iptables-alerter-xgbvx","kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal","openshift-multus/multus-7cxkg","openshift-multus/multus-additional-cni-plugins-cmrct","openshift-multus/network-metrics-daemon-hdvwv","openshift-network-diagnostics/network-check-target-v5zlw","openshift-ovn-kubernetes/ovnkube-node-lvfwt","kube-system/global-pull-secret-syncer-rzcl7"]
Jul 29 20:46:25.702934 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.702915    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xgbvx"
Jul 29 20:46:25.704972 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.704955    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:46:25.704972 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.704971    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 20:46:25.705125 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.704956    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 20:46:25.705125 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.705104    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hp4td\""
Jul 29 20:46:25.705240 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.705228    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.706602 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.706582    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 20:46:25.706865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.706845    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 20:46:25.706920 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.706879    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-snkvd\""
Jul 29 20:46:25.707209 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.707192    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 20:46:25.707318 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.707304    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 20:46:25.707385 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.707300    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 20:46:25.707385 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.707345    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 20:46:25.707635 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.707618    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.709294 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.709278    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fprvm\""
Jul 29 20:46:25.709294 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.709292    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 20:46:25.709418 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.709279    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:46:25.709781 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.709767    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2z8d"
Jul 29 20:46:25.711329 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.711315    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 20:46:25.711375 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.711338    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5wkml\""
Jul 29 20:46:25.711375 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.711364    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 20:46:25.711984 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.711972    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-562n4"
Jul 29 20:46:25.713437 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.713424    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 20:46:25.713721 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.713708    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 20:46:25.713863 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.713849    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vjl7x\""
Jul 29 20:46:25.713929 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.713853    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 20:46:25.713979 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.713938    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 20:46:25.714122 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.714108    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:25.714249 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.714222    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:25.716360 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.716343    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.718208 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.718191    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 20:46:25.718312 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.718224    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9w8sf\""
Jul 29 20:46:25.718379 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.718328    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 20:46:25.718379 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.718338    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 20:46:25.718740 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.718443    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 20:46:25.719300 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.718752    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m4fbv"
Jul 29 20:46:25.720321 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.720305    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 20:46:25.720368 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.720329    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 20:46:25.720400 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.720305    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-s4jsw\""
Jul 29 20:46:25.720700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.720656    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 20:41:24 +0000 UTC" deadline="2027-12-30 04:59:10.649591944 +0000 UTC"
Jul 29 20:46:25.720765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.720701    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12440h12m44.928894701s"
Jul 29 20:46:25.721746 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.721156    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.721294    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-systemd-units\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.721921    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-tuned\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.721959    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-run-k8s-cni-cncf-io\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.721998    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljwt\" (UniqueName: \"kubernetes.io/projected/562c3be5-ccad-4fbb-beaa-2403134cf790-kube-api-access-8ljwt\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-modprobe-d\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722168    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eeebe086-c70d-4f85-afbb-5e159fe76372-dbus\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722203    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/042d3857-ba4b-4d0d-b9da-3ad4d6003cd3-host\") pod \"node-ca-562n4\" (UID: \"042d3857-ba4b-4d0d-b9da-3ad4d6003cd3\") " pod="openshift-image-registry/node-ca-562n4"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722232    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-kubelet\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722274    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-run-netns\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722374    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-etc-openvswitch\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722412    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-run\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722446    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p9nc\" (UniqueName: \"kubernetes.io/projected/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-kube-api-access-5p9nc\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722479    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-system-cni-dir\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722516    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-os-release\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.722551 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722550    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-run-multus-certs\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722596    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-etc-kubernetes\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722625    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8pj\" (UniqueName: \"kubernetes.io/projected/df049e92-c4e8-449f-9dd1-8f14e7665d2b-kube-api-access-5p8pj\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722650    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-run-ovn-kubernetes\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722679    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-sys\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722710    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4z88\" (UniqueName: \"kubernetes.io/projected/4b75e3fd-cc2d-4da4-9026-c8b6302dae66-kube-api-access-n4z88\") pod \"node-resolver-g2z8d\" (UID: \"4b75e3fd-cc2d-4da4-9026-c8b6302dae66\") " pod="openshift-dns/node-resolver-g2z8d"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722739    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-var-lib-cni-bin\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722765    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-var-lib-kubelet\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722793    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-hostroot\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722823    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2xv6\" (UniqueName: \"kubernetes.io/projected/042d3857-ba4b-4d0d-b9da-3ad4d6003cd3-kube-api-access-t2xv6\") pod \"node-ca-562n4\" (UID: \"042d3857-ba4b-4d0d-b9da-3ad4d6003cd3\") " pod="openshift-image-registry/node-ca-562n4"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722854    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-var-lib-openvswitch\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722870    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722884    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-var-lib-kubelet\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722917    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df049e92-c4e8-449f-9dd1-8f14e7665d2b-cni-binary-copy\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722944    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-run-netns\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.722975    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpflw\" (UniqueName: \"kubernetes.io/projected/d4ed5571-9b79-4694-a064-6d75b38d7f3a-kube-api-access-mpflw\") pod \"iptables-alerter-xgbvx\" (UID: \"d4ed5571-9b79-4694-a064-6d75b38d7f3a\") " pod="openshift-network-operator/iptables-alerter-xgbvx"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723005    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-sysconfig\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.723372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723036    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-tmp\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723061    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eeebe086-c70d-4f85-afbb-5e159fe76372-kubelet-config\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723093    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-multus-conf-dir\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df049e92-c4e8-449f-9dd1-8f14e7665d2b-multus-daemon-config\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723180    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/042d3857-ba4b-4d0d-b9da-3ad4d6003cd3-serviceca\") pod \"node-ca-562n4\" (UID: \"042d3857-ba4b-4d0d-b9da-3ad4d6003cd3\") " pod="openshift-image-registry/node-ca-562n4"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723217    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-run-ovn\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723247    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-lib-modules\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723273    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4b75e3fd-cc2d-4da4-9026-c8b6302dae66-hosts-file\") pod \"node-resolver-g2z8d\" (UID: \"4b75e3fd-cc2d-4da4-9026-c8b6302dae66\") " pod="openshift-dns/node-resolver-g2z8d"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723304    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-multus-cni-dir\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723579    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-multus-socket-dir-parent\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723715    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723723    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/562c3be5-ccad-4fbb-beaa-2403134cf790-env-overrides\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723938    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723953    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7vbpq\""
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723972    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.723981    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-var-lib-cni-multus\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724029    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-systemd\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.724086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724062    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-cni-netd\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724111    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/562c3be5-ccad-4fbb-beaa-2403134cf790-ovnkube-config\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724165    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/562c3be5-ccad-4fbb-beaa-2403134cf790-ovn-node-metrics-cert\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724203    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-sysctl-d\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724266    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-log-socket\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724310    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-cni-bin\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724337    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-sysctl-conf\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724389    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4ed5571-9b79-4694-a064-6d75b38d7f3a-host-slash\") pod \"iptables-alerter-xgbvx\" (UID: \"d4ed5571-9b79-4694-a064-6d75b38d7f3a\") " pod="openshift-network-operator/iptables-alerter-xgbvx"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724424    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-run-systemd\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724476    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-run-openvswitch\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724501    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-node-log\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724539    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-kubernetes\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724619    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b75e3fd-cc2d-4da4-9026-c8b6302dae66-tmp-dir\") pod \"node-resolver-g2z8d\" (UID: \"4b75e3fd-cc2d-4da4-9026-c8b6302dae66\") " pod="openshift-dns/node-resolver-g2z8d"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724650    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-slash\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724717    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/562c3be5-ccad-4fbb-beaa-2403134cf790-ovnkube-script-lib\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724760    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-host\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.724842 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724827    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.725475 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724857    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-cnibin\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.725475 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.724879    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d4ed5571-9b79-4694-a064-6d75b38d7f3a-iptables-alerter-script\") pod \"iptables-alerter-xgbvx\" (UID: \"d4ed5571-9b79-4694-a064-6d75b38d7f3a\") " pod="openshift-network-operator/iptables-alerter-xgbvx"
Jul 29 20:46:25.725475 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.725310    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 20:46:25.725622 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.725501    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.727160 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.727126    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 20:46:25.727250 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.727159    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mcwvv\""
Jul 29 20:46:25.727250 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.727168    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 20:46:25.727788 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.727775    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:25.727868 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.727846    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:25.730024 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.729999    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:25.730096 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.730061    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:25.762973 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.762916    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-txn26"
Jul 29 20:46:25.770541 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.770518    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-txn26"
Jul 29 20:46:25.815180 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.815157    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 20:46:25.825832 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.825799    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d4ed5571-9b79-4694-a064-6d75b38d7f3a-iptables-alerter-script\") pod \"iptables-alerter-xgbvx\" (UID: \"d4ed5571-9b79-4694-a064-6d75b38d7f3a\") " pod="openshift-network-operator/iptables-alerter-xgbvx"
Jul 29 20:46:25.825832 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.825832    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-systemd-units\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.826029 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.825869    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-tuned\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.826029 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.825895    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-run-k8s-cni-cncf-io\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.826029 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.825924    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm8dw\" (UniqueName: \"kubernetes.io/projected/f4edb7a0-def8-488e-a6ea-eca10d6f1452-kube-api-access-rm8dw\") pod \"network-metrics-daemon-hdvwv\" (UID: \"f4edb7a0-def8-488e-a6ea-eca10d6f1452\") " pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:25.826029 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.825938    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-systemd-units\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.826029 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.825949    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.826029 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.825976    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljwt\" (UniqueName: \"kubernetes.io/projected/562c3be5-ccad-4fbb-beaa-2403134cf790-kube-api-access-8ljwt\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.826372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826067    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-modprobe-d\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.826372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826102    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eeebe086-c70d-4f85-afbb-5e159fe76372-dbus\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:25.826372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826218    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-modprobe-d\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.826372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826208    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 20:46:25.826372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826319    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-run-k8s-cni-cncf-io\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.826372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826319    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eeebe086-c70d-4f85-afbb-5e159fe76372-dbus\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:25.826372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826341    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-cnibin\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.826372 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826362    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/042d3857-ba4b-4d0d-b9da-3ad4d6003cd3-host\") pod \"node-ca-562n4\" (UID: \"042d3857-ba4b-4d0d-b9da-3ad4d6003cd3\") " pod="openshift-image-registry/node-ca-562n4"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826381    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-kubelet\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826396    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-run-netns\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826423    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-etc-openvswitch\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826420    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/042d3857-ba4b-4d0d-b9da-3ad4d6003cd3-host\") pod \"node-ca-562n4\" (UID: \"042d3857-ba4b-4d0d-b9da-3ad4d6003cd3\") " pod="openshift-image-registry/node-ca-562n4"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826442    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-run\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826469    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-run-netns\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826470    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-kubelet\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826470    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p9nc\" (UniqueName: \"kubernetes.io/projected/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-kube-api-access-5p9nc\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826510    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-run\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826525    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-system-cni-dir\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826478    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-etc-openvswitch\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826549    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-os-release\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826575    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-run-multus-certs\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826599    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-etc-kubernetes\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826602    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-system-cni-dir\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826613    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-os-release\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826621    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8pj\" (UniqueName: \"kubernetes.io/projected/df049e92-c4e8-449f-9dd1-8f14e7665d2b-kube-api-access-5p8pj\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.826661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826648    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-etc-kubernetes\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826648    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826657    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-run-multus-certs\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826683    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-run-ovn-kubernetes\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826718    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-sys\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826741    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-run-ovn-kubernetes\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4z88\" (UniqueName: \"kubernetes.io/projected/4b75e3fd-cc2d-4da4-9026-c8b6302dae66-kube-api-access-n4z88\") pod \"node-resolver-g2z8d\" (UID: \"4b75e3fd-cc2d-4da4-9026-c8b6302dae66\") " pod="openshift-dns/node-resolver-g2z8d"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826778    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-sys\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826787    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-var-lib-cni-bin\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-var-lib-kubelet\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826841    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-hostroot\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826863    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-var-lib-cni-bin\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826868    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826900    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2xv6\" (UniqueName: \"kubernetes.io/projected/042d3857-ba4b-4d0d-b9da-3ad4d6003cd3-kube-api-access-t2xv6\") pod \"node-ca-562n4\" (UID: \"042d3857-ba4b-4d0d-b9da-3ad4d6003cd3\") " pod="openshift-image-registry/node-ca-562n4"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826900    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-var-lib-kubelet\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826925    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-hostroot\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826935    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-var-lib-openvswitch\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.827350 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.826972    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-var-lib-kubelet\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827007    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d4ed5571-9b79-4694-a064-6d75b38d7f3a-iptables-alerter-script\") pod \"iptables-alerter-xgbvx\" (UID: \"d4ed5571-9b79-4694-a064-6d75b38d7f3a\") " pod="openshift-network-operator/iptables-alerter-xgbvx"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827033    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-var-lib-openvswitch\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827043    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df049e92-c4e8-449f-9dd1-8f14e7665d2b-cni-binary-copy\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827082    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-var-lib-kubelet\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827096    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-run-netns\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827124    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-socket-dir\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827183    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpflw\" (UniqueName: \"kubernetes.io/projected/d4ed5571-9b79-4694-a064-6d75b38d7f3a-kube-api-access-mpflw\") pod \"iptables-alerter-xgbvx\" (UID: \"d4ed5571-9b79-4694-a064-6d75b38d7f3a\") " pod="openshift-network-operator/iptables-alerter-xgbvx"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827209    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-sysconfig\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827183    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-run-netns\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827231    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-tmp\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827270    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eeebe086-c70d-4f85-afbb-5e159fe76372-kubelet-config\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827295    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-multus-conf-dir\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827318    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df049e92-c4e8-449f-9dd1-8f14e7665d2b-multus-daemon-config\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827342    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs\") pod \"network-metrics-daemon-hdvwv\" (UID: \"f4edb7a0-def8-488e-a6ea-eca10d6f1452\") " pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827368    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/042d3857-ba4b-4d0d-b9da-3ad4d6003cd3-serviceca\") pod \"node-ca-562n4\" (UID: \"042d3857-ba4b-4d0d-b9da-3ad4d6003cd3\") " pod="openshift-image-registry/node-ca-562n4"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827404    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-run-ovn\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.828171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827422    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-sysconfig\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827431    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-lib-modules\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827462    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4b75e3fd-cc2d-4da4-9026-c8b6302dae66-hosts-file\") pod \"node-resolver-g2z8d\" (UID: \"4b75e3fd-cc2d-4da4-9026-c8b6302dae66\") " pod="openshift-dns/node-resolver-g2z8d"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827471    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eeebe086-c70d-4f85-afbb-5e159fe76372-kubelet-config\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827484    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-multus-cni-dir\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827509    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-multus-conf-dir\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827515    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-multus-socket-dir-parent\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827552    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-run-ovn\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827558    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/562c3be5-ccad-4fbb-beaa-2403134cf790-env-overrides\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827599    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827621    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-var-lib-cni-multus\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827647    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc4k5\" (UniqueName: \"kubernetes.io/projected/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-kube-api-access-pc4k5\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827692    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm9g4\" (UniqueName: \"kubernetes.io/projected/ecbd1cee-616f-4c39-a16c-a806f1598b0f-kube-api-access-jm9g4\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827719    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-systemd\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827749    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827758    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df049e92-c4e8-449f-9dd1-8f14e7665d2b-cni-binary-copy\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827774    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-cni-netd\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.828969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827798    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/562c3be5-ccad-4fbb-beaa-2403134cf790-ovnkube-config\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827829    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/562c3be5-ccad-4fbb-beaa-2403134cf790-ovn-node-metrics-cert\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827858    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-sysctl-d\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827885    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-etc-selinux\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.827896    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827915    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-log-socket\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827932    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df049e92-c4e8-449f-9dd1-8f14e7665d2b-multus-daemon-config\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-cni-bin\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.827959    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret podName:eeebe086-c70d-4f85-afbb-5e159fe76372 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:26.327937441 +0000 UTC m=+1.924229117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret") pod "global-pull-secret-syncer-rzcl7" (UID: "eeebe086-c70d-4f85-afbb-5e159fe76372") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.827982    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-cni-bin\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828001    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-cni-netd\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828015    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4b75e3fd-cc2d-4da4-9026-c8b6302dae66-hosts-file\") pod \"node-resolver-g2z8d\" (UID: \"4b75e3fd-cc2d-4da4-9026-c8b6302dae66\") " pod="openshift-dns/node-resolver-g2z8d"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828003    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/042d3857-ba4b-4d0d-b9da-3ad4d6003cd3-serviceca\") pod \"node-ca-562n4\" (UID: \"042d3857-ba4b-4d0d-b9da-3ad4d6003cd3\") " pod="openshift-image-registry/node-ca-562n4"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828081    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-multus-cni-dir\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828109    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-host-var-lib-cni-multus\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828171    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-systemd\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828090    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-lib-modules\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.829702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828446    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-sysctl-conf\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828477    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67814655-dde0-40dc-b521-b4242f0cec15-agent-certs\") pod \"konnectivity-agent-m4fbv\" (UID: \"67814655-dde0-40dc-b521-b4242f0cec15\") " pod="kube-system/konnectivity-agent-m4fbv"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828510    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/562c3be5-ccad-4fbb-beaa-2403134cf790-env-overrides\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828521    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67814655-dde0-40dc-b521-b4242f0cec15-konnectivity-ca\") pod \"konnectivity-agent-m4fbv\" (UID: \"67814655-dde0-40dc-b521-b4242f0cec15\") " pod="kube-system/konnectivity-agent-m4fbv"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828539    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/562c3be5-ccad-4fbb-beaa-2403134cf790-ovnkube-config\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828548    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-device-dir\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828572    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-sys-fs\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828628    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-sysctl-d\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828665    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-multus-socket-dir-parent\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828705    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4ed5571-9b79-4694-a064-6d75b38d7f3a-host-slash\") pod \"iptables-alerter-xgbvx\" (UID: \"d4ed5571-9b79-4694-a064-6d75b38d7f3a\") " pod="openshift-network-operator/iptables-alerter-xgbvx"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828713    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-log-socket\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828766    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-run-systemd\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828805    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4ed5571-9b79-4694-a064-6d75b38d7f3a-host-slash\") pod \"iptables-alerter-xgbvx\" (UID: \"d4ed5571-9b79-4694-a064-6d75b38d7f3a\") " pod="openshift-network-operator/iptables-alerter-xgbvx"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828807    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-sysctl-conf\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828895    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-run-openvswitch\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828917    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-run-systemd\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828922    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-node-log\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.830232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828947    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-run-openvswitch\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828958    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-node-log\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.828975    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-kubernetes\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829006    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b75e3fd-cc2d-4da4-9026-c8b6302dae66-tmp-dir\") pod \"node-resolver-g2z8d\" (UID: \"4b75e3fd-cc2d-4da4-9026-c8b6302dae66\") " pod="openshift-dns/node-resolver-g2z8d"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829034    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-os-release\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829037    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-kubernetes\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829060    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829115    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-slash\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829139    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/562c3be5-ccad-4fbb-beaa-2403134cf790-ovnkube-script-lib\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829268    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-slash\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829297    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-host\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829327    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b75e3fd-cc2d-4da4-9026-c8b6302dae66-tmp-dir\") pod \"node-resolver-g2z8d\" (UID: \"4b75e3fd-cc2d-4da4-9026-c8b6302dae66\") " pod="openshift-dns/node-resolver-g2z8d"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-system-cni-dir\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829360    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwm82\" (UniqueName: \"kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82\") pod \"network-check-target-v5zlw\" (UID: \"c9d95b24-a79a-4f80-ad4c-34cde58ecf3d\") " pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829381    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-registration-dir\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829401    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-host\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829406    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.830765 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829449    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-cnibin\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.831289 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829517    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/562c3be5-ccad-4fbb-beaa-2403134cf790-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.831289 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829566    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df049e92-c4e8-449f-9dd1-8f14e7665d2b-cnibin\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.831289 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.829630    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/562c3be5-ccad-4fbb-beaa-2403134cf790-ovnkube-script-lib\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.831289 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.830046    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-tmp\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.831289 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.830072    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-etc-tuned\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.831289 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.831066    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/562c3be5-ccad-4fbb-beaa-2403134cf790-ovn-node-metrics-cert\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.841623 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.841306    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p9nc\" (UniqueName: \"kubernetes.io/projected/730ebef9-5750-4fc6-83b2-e7ad81f3bce7-kube-api-access-5p9nc\") pod \"tuned-wvq8v\" (UID: \"730ebef9-5750-4fc6-83b2-e7ad81f3bce7\") " pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:25.842378 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.842320    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpflw\" (UniqueName: \"kubernetes.io/projected/d4ed5571-9b79-4694-a064-6d75b38d7f3a-kube-api-access-mpflw\") pod \"iptables-alerter-xgbvx\" (UID: \"d4ed5571-9b79-4694-a064-6d75b38d7f3a\") " pod="openshift-network-operator/iptables-alerter-xgbvx"
Jul 29 20:46:25.842378 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.842341    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2xv6\" (UniqueName: \"kubernetes.io/projected/042d3857-ba4b-4d0d-b9da-3ad4d6003cd3-kube-api-access-t2xv6\") pod \"node-ca-562n4\" (UID: \"042d3857-ba4b-4d0d-b9da-3ad4d6003cd3\") " pod="openshift-image-registry/node-ca-562n4"
Jul 29 20:46:25.843077 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.843053    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljwt\" (UniqueName: \"kubernetes.io/projected/562c3be5-ccad-4fbb-beaa-2403134cf790-kube-api-access-8ljwt\") pod \"ovnkube-node-lvfwt\" (UID: \"562c3be5-ccad-4fbb-beaa-2403134cf790\") " pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:25.843437 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.843414    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8pj\" (UniqueName: \"kubernetes.io/projected/df049e92-c4e8-449f-9dd1-8f14e7665d2b-kube-api-access-5p8pj\") pod \"multus-7cxkg\" (UID: \"df049e92-c4e8-449f-9dd1-8f14e7665d2b\") " pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:25.843546 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.843529    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4z88\" (UniqueName: \"kubernetes.io/projected/4b75e3fd-cc2d-4da4-9026-c8b6302dae66-kube-api-access-n4z88\") pod \"node-resolver-g2z8d\" (UID: \"4b75e3fd-cc2d-4da4-9026-c8b6302dae66\") " pod="openshift-dns/node-resolver-g2z8d"
Jul 29 20:46:25.930306 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930276    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-os-release\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.930430 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930318    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.930491 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930438    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-system-cni-dir\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.930491 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930466    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-system-cni-dir\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.930491 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930472    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwm82\" (UniqueName: \"kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82\") pod \"network-check-target-v5zlw\" (UID: \"c9d95b24-a79a-4f80-ad4c-34cde58ecf3d\") " pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:25.930642 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930441    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-os-release\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.930642 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930495    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-registration-dir\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.930642 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930544    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-registration-dir\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.930642 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930541    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm8dw\" (UniqueName: \"kubernetes.io/projected/f4edb7a0-def8-488e-a6ea-eca10d6f1452-kube-api-access-rm8dw\") pod \"network-metrics-daemon-hdvwv\" (UID: \"f4edb7a0-def8-488e-a6ea-eca10d6f1452\") " pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:25.930642 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930582    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.930642 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930599    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-cnibin\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.930642 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.930974 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930659    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.930974 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930698    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.930974 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930726    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-cnibin\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.930974 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930775    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-socket-dir\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.930974 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930867    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.930974 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930904    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-socket-dir\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.930974 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930908    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.930974 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs\") pod \"network-metrics-daemon-hdvwv\" (UID: \"f4edb7a0-def8-488e-a6ea-eca10d6f1452\") " pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:25.931310 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.930979    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc4k5\" (UniqueName: \"kubernetes.io/projected/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-kube-api-access-pc4k5\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.931310 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.931026    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm9g4\" (UniqueName: \"kubernetes.io/projected/ecbd1cee-616f-4c39-a16c-a806f1598b0f-kube-api-access-jm9g4\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.931310 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.931056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.931310 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.931063    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:25.931310 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.931077    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-etc-selinux\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.931310 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.931102    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67814655-dde0-40dc-b521-b4242f0cec15-agent-certs\") pod \"konnectivity-agent-m4fbv\" (UID: \"67814655-dde0-40dc-b521-b4242f0cec15\") " pod="kube-system/konnectivity-agent-m4fbv"
Jul 29 20:46:25.931310 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.931120    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs podName:f4edb7a0-def8-488e-a6ea-eca10d6f1452 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:26.431101502 +0000 UTC m=+2.027393189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs") pod "network-metrics-daemon-hdvwv" (UID: "f4edb7a0-def8-488e-a6ea-eca10d6f1452") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:25.931310 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.931164    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.931310 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.931207    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67814655-dde0-40dc-b521-b4242f0cec15-konnectivity-ca\") pod \"konnectivity-agent-m4fbv\" (UID: \"67814655-dde0-40dc-b521-b4242f0cec15\") " pod="kube-system/konnectivity-agent-m4fbv"
Jul 29 20:46:25.931310 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.931238    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-device-dir\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.931310 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.931239    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-etc-selinux\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.931310 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.931262    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-sys-fs\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.931310 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.931301    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-device-dir\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.931750 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.931338    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ecbd1cee-616f-4c39-a16c-a806f1598b0f-sys-fs\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.931750 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.931509    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.931750 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.931664    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67814655-dde0-40dc-b521-b4242f0cec15-konnectivity-ca\") pod \"konnectivity-agent-m4fbv\" (UID: \"67814655-dde0-40dc-b521-b4242f0cec15\") " pod="kube-system/konnectivity-agent-m4fbv"
Jul 29 20:46:25.933375 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.933358    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67814655-dde0-40dc-b521-b4242f0cec15-agent-certs\") pod \"konnectivity-agent-m4fbv\" (UID: \"67814655-dde0-40dc-b521-b4242f0cec15\") " pod="kube-system/konnectivity-agent-m4fbv"
Jul 29 20:46:25.938577 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.938552    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:46:25.938694 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.938580    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:46:25.938694 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.938596    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lwm82 for pod openshift-network-diagnostics/network-check-target-v5zlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:25.938694 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:25.938662    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82 podName:c9d95b24-a79a-4f80-ad4c-34cde58ecf3d nodeName:}" failed. No retries permitted until 2026-07-29 20:46:26.438641178 +0000 UTC m=+2.034932839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lwm82" (UniqueName: "kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82") pod "network-check-target-v5zlw" (UID: "c9d95b24-a79a-4f80-ad4c-34cde58ecf3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:25.939787 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:25.939725    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e50be1114c5d989169bea87a3d8b446.slice/crio-f54794cdff3ae30c26b35e8d86946dbc45ab838029cebbf3e6016f269cf4954a WatchSource:0}: Error finding container f54794cdff3ae30c26b35e8d86946dbc45ab838029cebbf3e6016f269cf4954a: Status 404 returned error can't find the container with id f54794cdff3ae30c26b35e8d86946dbc45ab838029cebbf3e6016f269cf4954a
Jul 29 20:46:25.940159 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:25.940125    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5318fce92e6dc30124e077fbe98465.slice/crio-2a8786a6f03d8f51a7a23d42a4e3a6255eb5444e0b32449b5eab7741fac26964 WatchSource:0}: Error finding container 2a8786a6f03d8f51a7a23d42a4e3a6255eb5444e0b32449b5eab7741fac26964: Status 404 returned error can't find the container with id 2a8786a6f03d8f51a7a23d42a4e3a6255eb5444e0b32449b5eab7741fac26964
Jul 29 20:46:25.941501 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.941476    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc4k5\" (UniqueName: \"kubernetes.io/projected/c4bb2f8a-22d1-4fd5-9722-81d47ada14f1-kube-api-access-pc4k5\") pod \"multus-additional-cni-plugins-cmrct\" (UID: \"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1\") " pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:25.941709 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.941688    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm8dw\" (UniqueName: \"kubernetes.io/projected/f4edb7a0-def8-488e-a6ea-eca10d6f1452-kube-api-access-rm8dw\") pod \"network-metrics-daemon-hdvwv\" (UID: \"f4edb7a0-def8-488e-a6ea-eca10d6f1452\") " pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:25.941755 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.941738    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm9g4\" (UniqueName: \"kubernetes.io/projected/ecbd1cee-616f-4c39-a16c-a806f1598b0f-kube-api-access-jm9g4\") pod \"aws-ebs-csi-driver-node-w85sm\" (UID: \"ecbd1cee-616f-4c39-a16c-a806f1598b0f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:25.944881 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:25.944862    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:46:26.031273 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.031184    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xgbvx"
Jul 29 20:46:26.040868 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:26.040842    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ed5571_9b79_4694_a064_6d75b38d7f3a.slice/crio-8b744f87dfa7fc88f5db325ccd2a1ee9b5f6d1bec7356c856116bf515b8930af WatchSource:0}: Error finding container 8b744f87dfa7fc88f5db325ccd2a1ee9b5f6d1bec7356c856116bf515b8930af: Status 404 returned error can't find the container with id 8b744f87dfa7fc88f5db325ccd2a1ee9b5f6d1bec7356c856116bf515b8930af
Jul 29 20:46:26.048131 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.048111    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:26.056853 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:26.056829    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562c3be5_ccad_4fbb_beaa_2403134cf790.slice/crio-e38d50c2f728fad3f32dfd04fb181a530b16c9dba9d8c4a86e78c5bdf2217809 WatchSource:0}: Error finding container e38d50c2f728fad3f32dfd04fb181a530b16c9dba9d8c4a86e78c5bdf2217809: Status 404 returned error can't find the container with id e38d50c2f728fad3f32dfd04fb181a530b16c9dba9d8c4a86e78c5bdf2217809
Jul 29 20:46:26.068879 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.068854    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wvq8v"
Jul 29 20:46:26.078302 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:26.078275    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730ebef9_5750_4fc6_83b2_e7ad81f3bce7.slice/crio-8706a984da74d032497cbc3536fb4a76bb4db1f59efc000e44f8f74f0da197cb WatchSource:0}: Error finding container 8706a984da74d032497cbc3536fb4a76bb4db1f59efc000e44f8f74f0da197cb: Status 404 returned error can't find the container with id 8706a984da74d032497cbc3536fb4a76bb4db1f59efc000e44f8f74f0da197cb
Jul 29 20:46:26.088180 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.088159    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2z8d"
Jul 29 20:46:26.096190 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:26.096167    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b75e3fd_cc2d_4da4_9026_c8b6302dae66.slice/crio-9e40b44a85fc2582c3a2faffcc10cd4f614c6ffd6f9bae465141832ba5582c4b WatchSource:0}: Error finding container 9e40b44a85fc2582c3a2faffcc10cd4f614c6ffd6f9bae465141832ba5582c4b: Status 404 returned error can't find the container with id 9e40b44a85fc2582c3a2faffcc10cd4f614c6ffd6f9bae465141832ba5582c4b
Jul 29 20:46:26.102155 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.102117    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-562n4"
Jul 29 20:46:26.110626 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:26.110593    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042d3857_ba4b_4d0d_b9da_3ad4d6003cd3.slice/crio-ffcf97d6b4e7214ecf0adbacb64bdc24e798cbc060fef1708d11c45699fc404f WatchSource:0}: Error finding container ffcf97d6b4e7214ecf0adbacb64bdc24e798cbc060fef1708d11c45699fc404f: Status 404 returned error can't find the container with id ffcf97d6b4e7214ecf0adbacb64bdc24e798cbc060fef1708d11c45699fc404f
Jul 29 20:46:26.119136 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.119114    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7cxkg"
Jul 29 20:46:26.127277 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:26.127249    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf049e92_c4e8_449f_9dd1_8f14e7665d2b.slice/crio-bd5966c52c3d92f487f5b4e974ee972f10131ad5decbdd83cf306f2fde44969a WatchSource:0}: Error finding container bd5966c52c3d92f487f5b4e974ee972f10131ad5decbdd83cf306f2fde44969a: Status 404 returned error can't find the container with id bd5966c52c3d92f487f5b4e974ee972f10131ad5decbdd83cf306f2fde44969a
Jul 29 20:46:26.142863 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.142840    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m4fbv"
Jul 29 20:46:26.151235 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:26.151214    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67814655_dde0_40dc_b521_b4242f0cec15.slice/crio-8787cd5bf1510fb563d0e7b3ddfa445bb70db2ba861a7103033fc7a1ab57d4e1 WatchSource:0}: Error finding container 8787cd5bf1510fb563d0e7b3ddfa445bb70db2ba861a7103033fc7a1ab57d4e1: Status 404 returned error can't find the container with id 8787cd5bf1510fb563d0e7b3ddfa445bb70db2ba861a7103033fc7a1ab57d4e1
Jul 29 20:46:26.163070 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.163045    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm"
Jul 29 20:46:26.167762 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.167743    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cmrct"
Jul 29 20:46:26.172495 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:26.172473    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbd1cee_616f_4c39_a16c_a806f1598b0f.slice/crio-a570a6181f5ad7e8dd57ebebed55efe8950d950d0a4778d515825562de7d117a WatchSource:0}: Error finding container a570a6181f5ad7e8dd57ebebed55efe8950d950d0a4778d515825562de7d117a: Status 404 returned error can't find the container with id a570a6181f5ad7e8dd57ebebed55efe8950d950d0a4778d515825562de7d117a
Jul 29 20:46:26.176500 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:46:26.176473    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb2f8a_22d1_4fd5_9722_81d47ada14f1.slice/crio-de9f34551d1e365cb7b0e1eaa94bd8bec8d05453f04c9fe5646d6025eb347ab0 WatchSource:0}: Error finding container de9f34551d1e365cb7b0e1eaa94bd8bec8d05453f04c9fe5646d6025eb347ab0: Status 404 returned error can't find the container with id de9f34551d1e365cb7b0e1eaa94bd8bec8d05453f04c9fe5646d6025eb347ab0
Jul 29 20:46:26.333682 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.333506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:26.333846 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:26.333718    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:26.333846 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:26.333784    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret podName:eeebe086-c70d-4f85-afbb-5e159fe76372 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:27.333764943 +0000 UTC m=+2.930056626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret") pod "global-pull-secret-syncer-rzcl7" (UID: "eeebe086-c70d-4f85-afbb-5e159fe76372") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:26.434138 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.434101    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs\") pod \"network-metrics-daemon-hdvwv\" (UID: \"f4edb7a0-def8-488e-a6ea-eca10d6f1452\") " pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:26.434331 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:26.434273    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:26.434400 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:26.434354    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs podName:f4edb7a0-def8-488e-a6ea-eca10d6f1452 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:27.434317361 +0000 UTC m=+3.030609038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs") pod "network-metrics-daemon-hdvwv" (UID: "f4edb7a0-def8-488e-a6ea-eca10d6f1452") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:26.535078 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.535042    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwm82\" (UniqueName: \"kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82\") pod \"network-check-target-v5zlw\" (UID: \"c9d95b24-a79a-4f80-ad4c-34cde58ecf3d\") " pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:26.535314 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:26.535295    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:46:26.535408 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:26.535326    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:46:26.535408 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:26.535340    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lwm82 for pod openshift-network-diagnostics/network-check-target-v5zlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:26.535408 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:26.535401    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82 podName:c9d95b24-a79a-4f80-ad4c-34cde58ecf3d nodeName:}" failed. No retries permitted until 2026-07-29 20:46:27.53538115 +0000 UTC m=+3.131672817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lwm82" (UniqueName: "kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82") pod "network-check-target-v5zlw" (UID: "c9d95b24-a79a-4f80-ad4c-34cde58ecf3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:26.773840 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.773728    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 20:41:25 +0000 UTC" deadline="2028-04-29 22:42:15.252134946 +0000 UTC"
Jul 29 20:46:26.773840 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.773769    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15361h55m48.478371106s"
Jul 29 20:46:26.872444 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.872375    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrct" event={"ID":"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1","Type":"ContainerStarted","Data":"de9f34551d1e365cb7b0e1eaa94bd8bec8d05453f04c9fe5646d6025eb347ab0"}
Jul 29 20:46:26.880688 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.880658    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:46:26.883995 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.883906    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7cxkg" event={"ID":"df049e92-c4e8-449f-9dd1-8f14e7665d2b","Type":"ContainerStarted","Data":"bd5966c52c3d92f487f5b4e974ee972f10131ad5decbdd83cf306f2fde44969a"}
Jul 29 20:46:26.897479 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.897395    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-562n4" event={"ID":"042d3857-ba4b-4d0d-b9da-3ad4d6003cd3","Type":"ContainerStarted","Data":"ffcf97d6b4e7214ecf0adbacb64bdc24e798cbc060fef1708d11c45699fc404f"}
Jul 29 20:46:26.911708 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.911664    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wvq8v" event={"ID":"730ebef9-5750-4fc6-83b2-e7ad81f3bce7","Type":"ContainerStarted","Data":"8706a984da74d032497cbc3536fb4a76bb4db1f59efc000e44f8f74f0da197cb"}
Jul 29 20:46:26.939852 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.939812    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt" event={"ID":"562c3be5-ccad-4fbb-beaa-2403134cf790","Type":"ContainerStarted","Data":"e38d50c2f728fad3f32dfd04fb181a530b16c9dba9d8c4a86e78c5bdf2217809"}
Jul 29 20:46:26.964933 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.964894    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xgbvx" event={"ID":"d4ed5571-9b79-4694-a064-6d75b38d7f3a","Type":"ContainerStarted","Data":"8b744f87dfa7fc88f5db325ccd2a1ee9b5f6d1bec7356c856116bf515b8930af"}
Jul 29 20:46:26.974422 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.974350    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal" event={"ID":"9e50be1114c5d989169bea87a3d8b446","Type":"ContainerStarted","Data":"f54794cdff3ae30c26b35e8d86946dbc45ab838029cebbf3e6016f269cf4954a"}
Jul 29 20:46:26.977646 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.977543    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal" event={"ID":"5d5318fce92e6dc30124e077fbe98465","Type":"ContainerStarted","Data":"2a8786a6f03d8f51a7a23d42a4e3a6255eb5444e0b32449b5eab7741fac26964"}
Jul 29 20:46:26.985611 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.985370    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm" event={"ID":"ecbd1cee-616f-4c39-a16c-a806f1598b0f","Type":"ContainerStarted","Data":"a570a6181f5ad7e8dd57ebebed55efe8950d950d0a4778d515825562de7d117a"}
Jul 29 20:46:26.994102 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:26.993882    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m4fbv" event={"ID":"67814655-dde0-40dc-b521-b4242f0cec15","Type":"ContainerStarted","Data":"8787cd5bf1510fb563d0e7b3ddfa445bb70db2ba861a7103033fc7a1ab57d4e1"}
Jul 29 20:46:27.021381 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:27.021339    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2z8d" event={"ID":"4b75e3fd-cc2d-4da4-9026-c8b6302dae66","Type":"ContainerStarted","Data":"9e40b44a85fc2582c3a2faffcc10cd4f614c6ffd6f9bae465141832ba5582c4b"}
Jul 29 20:46:27.344320 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:27.344284    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:27.344507 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:27.344476    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:27.344583 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:27.344540    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret podName:eeebe086-c70d-4f85-afbb-5e159fe76372 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:29.344521725 +0000 UTC m=+4.940813403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret") pod "global-pull-secret-syncer-rzcl7" (UID: "eeebe086-c70d-4f85-afbb-5e159fe76372") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:27.444876 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:27.444834    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs\") pod \"network-metrics-daemon-hdvwv\" (UID: \"f4edb7a0-def8-488e-a6ea-eca10d6f1452\") " pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:27.445065 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:27.444980    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:27.445065 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:27.445045    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs podName:f4edb7a0-def8-488e-a6ea-eca10d6f1452 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:29.445024465 +0000 UTC m=+5.041316143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs") pod "network-metrics-daemon-hdvwv" (UID: "f4edb7a0-def8-488e-a6ea-eca10d6f1452") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:27.545757 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:27.545714    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwm82\" (UniqueName: \"kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82\") pod \"network-check-target-v5zlw\" (UID: \"c9d95b24-a79a-4f80-ad4c-34cde58ecf3d\") " pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:27.545947 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:27.545905    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:46:27.545947 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:27.545923    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:46:27.545947 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:27.545936    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lwm82 for pod openshift-network-diagnostics/network-check-target-v5zlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:27.546125 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:27.545992    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82 podName:c9d95b24-a79a-4f80-ad4c-34cde58ecf3d nodeName:}" failed. No retries permitted until 2026-07-29 20:46:29.545973991 +0000 UTC m=+5.142265668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lwm82" (UniqueName: "kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82") pod "network-check-target-v5zlw" (UID: "c9d95b24-a79a-4f80-ad4c-34cde58ecf3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:27.775095 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:27.774966    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 20:41:25 +0000 UTC" deadline="2028-04-29 00:47:21.594414412 +0000 UTC"
Jul 29 20:46:27.775095 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:27.775046    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15340h0m53.819372262s"
Jul 29 20:46:27.859887 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:27.859851    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:27.860052 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:27.860007    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:27.860456 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:27.860437    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:27.860574 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:27.860554    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:27.860640 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:27.860619    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:27.860715 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:27.860698    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:28.011662 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:28.011628    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:46:29.368442 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:29.367805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:29.368442 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:29.368009    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:29.368442 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:29.368074    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret podName:eeebe086-c70d-4f85-afbb-5e159fe76372 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:33.368055164 +0000 UTC m=+8.964346831 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret") pod "global-pull-secret-syncer-rzcl7" (UID: "eeebe086-c70d-4f85-afbb-5e159fe76372") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:29.468948 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:29.468910    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs\") pod \"network-metrics-daemon-hdvwv\" (UID: \"f4edb7a0-def8-488e-a6ea-eca10d6f1452\") " pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:29.469123 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:29.469095    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:29.469238 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:29.469175    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs podName:f4edb7a0-def8-488e-a6ea-eca10d6f1452 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:33.469155648 +0000 UTC m=+9.065447323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs") pod "network-metrics-daemon-hdvwv" (UID: "f4edb7a0-def8-488e-a6ea-eca10d6f1452") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:29.569434 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:29.569402    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwm82\" (UniqueName: \"kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82\") pod \"network-check-target-v5zlw\" (UID: \"c9d95b24-a79a-4f80-ad4c-34cde58ecf3d\") " pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:29.569649 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:29.569587    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:46:29.569649 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:29.569610    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:46:29.569649 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:29.569624    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lwm82 for pod openshift-network-diagnostics/network-check-target-v5zlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:29.569803 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:29.569688    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82 podName:c9d95b24-a79a-4f80-ad4c-34cde58ecf3d nodeName:}" failed. No retries permitted until 2026-07-29 20:46:33.569668688 +0000 UTC m=+9.165960364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lwm82" (UniqueName: "kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82") pod "network-check-target-v5zlw" (UID: "c9d95b24-a79a-4f80-ad4c-34cde58ecf3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:29.860184 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:29.860136    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:29.860385 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:29.860288    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:29.860696 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:29.860672    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:29.860811 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:29.860777    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:29.860868 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:29.860833    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:29.860933 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:29.860915    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:31.859377 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:31.859338    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:31.859831 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:31.859339    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:31.859831 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:31.859477    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:31.859831 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:31.859511    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:31.859831 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:31.859656    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:31.859831 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:31.859750    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:33.403693 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:33.403658    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:33.404243 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:33.403796    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:33.404243 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:33.403855    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret podName:eeebe086-c70d-4f85-afbb-5e159fe76372 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:41.403837628 +0000 UTC m=+17.000129305 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret") pod "global-pull-secret-syncer-rzcl7" (UID: "eeebe086-c70d-4f85-afbb-5e159fe76372") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:33.504233 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:33.504198    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs\") pod \"network-metrics-daemon-hdvwv\" (UID: \"f4edb7a0-def8-488e-a6ea-eca10d6f1452\") " pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:33.504426 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:33.504375    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:33.504504 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:33.504456    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs podName:f4edb7a0-def8-488e-a6ea-eca10d6f1452 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:41.504434498 +0000 UTC m=+17.100726179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs") pod "network-metrics-daemon-hdvwv" (UID: "f4edb7a0-def8-488e-a6ea-eca10d6f1452") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:33.604764 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:33.604722    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwm82\" (UniqueName: \"kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82\") pod \"network-check-target-v5zlw\" (UID: \"c9d95b24-a79a-4f80-ad4c-34cde58ecf3d\") " pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:33.604946 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:33.604881    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:46:33.604946 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:33.604899    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:46:33.604946 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:33.604912    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lwm82 for pod openshift-network-diagnostics/network-check-target-v5zlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:33.605115 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:33.604968    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82 podName:c9d95b24-a79a-4f80-ad4c-34cde58ecf3d nodeName:}" failed. No retries permitted until 2026-07-29 20:46:41.604950311 +0000 UTC m=+17.201241976 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lwm82" (UniqueName: "kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82") pod "network-check-target-v5zlw" (UID: "c9d95b24-a79a-4f80-ad4c-34cde58ecf3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:33.859508 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:33.859471    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:33.859689 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:33.859620    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:33.859689 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:33.859472    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:33.859804 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:33.859729    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:33.859804 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:33.859472    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:33.859919 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:33.859806    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:35.859947 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:35.859896    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:35.859947 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:35.859940    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:35.860537 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:35.860049    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:35.860537 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:35.860087    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:35.860537 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:35.860210    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:35.860537 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:35.860308    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:37.860072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:37.860025    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:37.860072 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:37.860073    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:37.860601 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:37.860073    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:37.860601 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:37.860180    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:37.860601 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:37.860251    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:37.860601 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:37.860303    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:39.860108 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:39.860068    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:39.860598 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:39.860071    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:39.860598 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:39.860209    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:39.860598 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:39.860071    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:39.860598 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:39.860274    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:39.860598 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:39.860350    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:41.463660 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:41.463621    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:41.464074 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:41.463777    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:41.464074 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:41.463837    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret podName:eeebe086-c70d-4f85-afbb-5e159fe76372 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:57.463822378 +0000 UTC m=+33.060114045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret") pod "global-pull-secret-syncer-rzcl7" (UID: "eeebe086-c70d-4f85-afbb-5e159fe76372") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:41.564213 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:41.564175    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs\") pod \"network-metrics-daemon-hdvwv\" (UID: \"f4edb7a0-def8-488e-a6ea-eca10d6f1452\") " pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:41.564405 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:41.564308    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:41.564405 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:41.564385    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs podName:f4edb7a0-def8-488e-a6ea-eca10d6f1452 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:57.564365165 +0000 UTC m=+33.160656840 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs") pod "network-metrics-daemon-hdvwv" (UID: "f4edb7a0-def8-488e-a6ea-eca10d6f1452") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:41.665414 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:41.665375    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwm82\" (UniqueName: \"kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82\") pod \"network-check-target-v5zlw\" (UID: \"c9d95b24-a79a-4f80-ad4c-34cde58ecf3d\") " pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:41.665559 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:41.665538    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:46:41.665608 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:41.665562    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:46:41.665608 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:41.665572    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lwm82 for pod openshift-network-diagnostics/network-check-target-v5zlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:41.665674 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:41.665623    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82 podName:c9d95b24-a79a-4f80-ad4c-34cde58ecf3d nodeName:}" failed. No retries permitted until 2026-07-29 20:46:57.665609511 +0000 UTC m=+33.261901174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lwm82" (UniqueName: "kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82") pod "network-check-target-v5zlw" (UID: "c9d95b24-a79a-4f80-ad4c-34cde58ecf3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:41.859525 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:41.859487    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:41.859764 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:41.859530    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:41.859764 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:41.859533    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:41.859764 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:41.859617    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:41.859764 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:41.859721    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:41.859967 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:41.859800    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:43.860238 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:43.860203    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:43.860238 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:43.860234    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:43.860808 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:43.860234    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:43.860808 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:43.860312    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:43.860808 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:43.860420    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:43.860808 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:43.860500    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:45.859605 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:45.859567    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:45.859605 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:45.859587    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:45.860154 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:45.859587    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:45.860154 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:45.859671    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:45.860154 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:45.859741    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:45.860154 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:45.859834    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:46.079307 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:46.079273    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7cxkg" event={"ID":"df049e92-c4e8-449f-9dd1-8f14e7665d2b","Type":"ContainerStarted","Data":"5301bc6654ac248c5957affdb9bcf4dcd55eb6b13881595bb98519030d709816"}
Jul 29 20:46:46.082580 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:46.082543    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wvq8v" event={"ID":"730ebef9-5750-4fc6-83b2-e7ad81f3bce7","Type":"ContainerStarted","Data":"8e97d72f7798a105f4f3e18954752e8da6a11b1f1de70c16e8997409a3dacd89"}
Jul 29 20:46:46.086564 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:46.086473    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt" event={"ID":"562c3be5-ccad-4fbb-beaa-2403134cf790","Type":"ContainerStarted","Data":"2e44b76c80021d6b751aca6ab10c4f0c1722d97b5d60861d2da7a6c605beddfd"}
Jul 29 20:46:46.090760 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:46.090726    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal" event={"ID":"5d5318fce92e6dc30124e077fbe98465","Type":"ContainerStarted","Data":"29709147bef4109c7faabeacf22bc146c4b9f516e058e4bbbb660f9e89e22aff"}
Jul 29 20:46:46.109903 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:46.109546    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7cxkg" podStartSLOduration=2.623897619 podStartE2EDuration="22.109522011s" totalImagesPullingTime="19.485624392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:46.094664112 +0000 UTC m=+21.690955798" watchObservedRunningTime="2026-07-29 20:46:46.109522011 +0000 UTC m=+21.705813696"
Jul 29 20:46:46.110548 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:46.110309    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wvq8v" podStartSLOduration=2.608482231 podStartE2EDuration="22.110294719s" totalImagesPullingTime="19.501812488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:46.109508797 +0000 UTC m=+21.705800482" watchObservedRunningTime="2026-07-29 20:46:46.110294719 +0000 UTC m=+21.706586404"
Jul 29 20:46:46.122802 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:46.122586    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-194.ec2.internal" podStartSLOduration=21.122568365 podStartE2EDuration="21.122568365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:46.121702462 +0000 UTC m=+21.717994147" watchObservedRunningTime="2026-07-29 20:46:46.122568365 +0000 UTC m=+21.718860048"
Jul 29 20:46:47.094774 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.094570    2584 generic.go:356] "Generic (PLEG): container finished" podID="c4bb2f8a-22d1-4fd5-9722-81d47ada14f1" containerID="f5aae495a1b1b39ea3a53c4a2df41b3930bc804dd113eb64cce320cb0a149b28" exitCode=0
Jul 29 20:46:47.095603 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.094655    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrct" event={"ID":"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1","Type":"ContainerDied","Data":"f5aae495a1b1b39ea3a53c4a2df41b3930bc804dd113eb64cce320cb0a149b28"}
Jul 29 20:46:47.096631 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.096608    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-562n4" event={"ID":"042d3857-ba4b-4d0d-b9da-3ad4d6003cd3","Type":"ContainerStarted","Data":"fea360dc48762b9ed061cffa1ac51f4e135da5a84f3e5a9c61e20c4a11a5fd6f"}
Jul 29 20:46:47.100061 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.100037    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt" event={"ID":"562c3be5-ccad-4fbb-beaa-2403134cf790","Type":"ContainerStarted","Data":"d8e3775e7ca9c2046886952e8e475ac773957a6ad7d8aa96d7464edcd64e911c"}
Jul 29 20:46:47.100185 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.100113    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt" event={"ID":"562c3be5-ccad-4fbb-beaa-2403134cf790","Type":"ContainerStarted","Data":"637cea1b354a012493f0e4ef58372e3e8b3de405afb9bda2a1a9e414f8260053"}
Jul 29 20:46:47.100185 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.100176    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt" event={"ID":"562c3be5-ccad-4fbb-beaa-2403134cf790","Type":"ContainerStarted","Data":"b4efc94d80dd60da39a93087ff0208ff91f5150354ecb5469846c6794f72c960"}
Jul 29 20:46:47.100295 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.100211    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt" event={"ID":"562c3be5-ccad-4fbb-beaa-2403134cf790","Type":"ContainerStarted","Data":"8d5e4bfadd512109576463b16cf26ad2b4dec88b7718f416ba413e7a595f7577"}
Jul 29 20:46:47.100295 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.100240    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt" event={"ID":"562c3be5-ccad-4fbb-beaa-2403134cf790","Type":"ContainerStarted","Data":"dd25eb7baf85c0b0db5ed09bdf868719525ffe71b8decbf25942026f1add32fb"}
Jul 29 20:46:47.101628 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.101600    2584 generic.go:356] "Generic (PLEG): container finished" podID="9e50be1114c5d989169bea87a3d8b446" containerID="bb5382fbf823a24e3f7ca187d2494d5104f07ed3cf0f6f6bd870ca37a3332337" exitCode=0
Jul 29 20:46:47.101718 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.101671    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal" event={"ID":"9e50be1114c5d989169bea87a3d8b446","Type":"ContainerDied","Data":"bb5382fbf823a24e3f7ca187d2494d5104f07ed3cf0f6f6bd870ca37a3332337"}
Jul 29 20:46:47.103197 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.103175    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm" event={"ID":"ecbd1cee-616f-4c39-a16c-a806f1598b0f","Type":"ContainerStarted","Data":"997056000fb6bf84b646ef7d9c9079a9c2d40ec7f2566bf7ef04a13169487a10"}
Jul 29 20:46:47.104622 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.104601    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m4fbv" event={"ID":"67814655-dde0-40dc-b521-b4242f0cec15","Type":"ContainerStarted","Data":"dbb8b9070fdefb84f301269c8155fdb0b087a6c14ac9861b71d994445d5fbfe2"}
Jul 29 20:46:47.106112 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.106088    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2z8d" event={"ID":"4b75e3fd-cc2d-4da4-9026-c8b6302dae66","Type":"ContainerStarted","Data":"b7adee4cbcc6a3bcfcae9c4218d0b5c412a6db10bc20d61b370dafd68b2b1322"}
Jul 29 20:46:47.128266 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.128221    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m4fbv" podStartSLOduration=11.542979972 podStartE2EDuration="23.128205608s" totalImagesPullingTime="11.585225636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:47.127951532 +0000 UTC m=+22.724243240" watchObservedRunningTime="2026-07-29 20:46:47.128205608 +0000 UTC m=+22.724497329"
Jul 29 20:46:47.143414 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.143357    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g2z8d" podStartSLOduration=3.660442139 podStartE2EDuration="23.143335953s" totalImagesPullingTime="19.482893814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:47.14281744 +0000 UTC m=+22.739109149" watchObservedRunningTime="2026-07-29 20:46:47.143335953 +0000 UTC m=+22.739627640"
Jul 29 20:46:47.169338 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.169285    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-562n4" podStartSLOduration=3.701224082 podStartE2EDuration="23.169269105s" totalImagesPullingTime="19.468045023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:47.16896859 +0000 UTC m=+22.765260274" watchObservedRunningTime="2026-07-29 20:46:47.169269105 +0000 UTC m=+22.765560789"
Jul 29 20:46:47.717807 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.717766    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 20:46:47.791215 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.791043    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T20:46:47.71779951Z","UUID":"22f54cfc-2f3f-47cc-8f5a-5742b3f36511","Handler":null,"Name":"","Endpoint":""}
Jul 29 20:46:47.792695 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.792671    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 20:46:47.792812 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.792719    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 20:46:47.859816 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.859721    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:47.859816 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.859758    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:47.860042 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:47.859830    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:47.860042 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:47.859846    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:47.860042 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:47.859931    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:47.860042 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:47.860001    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:48.110589 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:48.110489    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xgbvx" event={"ID":"d4ed5571-9b79-4694-a064-6d75b38d7f3a","Type":"ContainerStarted","Data":"f5329ccc67b5cadaa47097191334be39ba6c6cebe8aaac4237e2c6ff57d6bd04"}
Jul 29 20:46:48.113447 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:48.113414    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal" event={"ID":"9e50be1114c5d989169bea87a3d8b446","Type":"ContainerStarted","Data":"822ced107d8377e5be247801987dad8cec5885c2da943c2edc3116404725392e"}
Jul 29 20:46:48.116028 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:48.115997    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm" event={"ID":"ecbd1cee-616f-4c39-a16c-a806f1598b0f","Type":"ContainerStarted","Data":"deae48f9bea649128b405498b3102b73c6216bc0eb0ebefd0251c0b5c0ca3404"}
Jul 29 20:46:48.123970 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:48.123912    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xgbvx" podStartSLOduration=4.586174041 podStartE2EDuration="24.123893168s" totalImagesPullingTime="19.537719127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:48.123054746 +0000 UTC m=+23.719346452" watchObservedRunningTime="2026-07-29 20:46:48.123893168 +0000 UTC m=+23.720184853"
Jul 29 20:46:48.137073 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:48.137019    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-194.ec2.internal" podStartSLOduration=23.137003234 podStartE2EDuration="23.137003234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:48.136184078 +0000 UTC m=+23.732475788" watchObservedRunningTime="2026-07-29 20:46:48.137003234 +0000 UTC m=+23.733294917"
Jul 29 20:46:49.122115 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:49.122073    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt" event={"ID":"562c3be5-ccad-4fbb-beaa-2403134cf790","Type":"ContainerStarted","Data":"db209edef84488acd79e5325cb560feb7c313330aed99fe947c98692c1a63e6f"}
Jul 29 20:46:49.124388 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:49.124350    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm" event={"ID":"ecbd1cee-616f-4c39-a16c-a806f1598b0f","Type":"ContainerStarted","Data":"5860638f7f3130564ce8388aabcd6c3d660ccd73c55fa970e4e0c013016adbcd"}
Jul 29 20:46:49.143047 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:49.142991    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w85sm" podStartSLOduration=3.276131898 podStartE2EDuration="25.142974247s" totalImagesPullingTime="21.866842349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:49.142553349 +0000 UTC m=+24.738845059" watchObservedRunningTime="2026-07-29 20:46:49.142974247 +0000 UTC m=+24.739265930"
Jul 29 20:46:49.374118 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:49.374041    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m4fbv"
Jul 29 20:46:49.374675 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:49.374652    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m4fbv"
Jul 29 20:46:49.859718 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:49.859673    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:49.859718 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:49.859720    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:49.859961 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:49.859789    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:49.859961 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:49.859808    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:49.859961 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:49.859913    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:49.860117 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:49.859989    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:50.127437 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:50.127296    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m4fbv"
Jul 29 20:46:50.127808 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:50.127617    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m4fbv"
Jul 29 20:46:51.860190 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:51.859985    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:51.860802 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:51.860001    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:51.860802 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:51.859985    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:51.860802 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:51.860305    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:51.860802 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:51.860411    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:51.860802 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:51.860484    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:52.132698 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:52.132618    2584 generic.go:356] "Generic (PLEG): container finished" podID="c4bb2f8a-22d1-4fd5-9722-81d47ada14f1" containerID="0903ec85a81a2ae25782eca65d6458568abbcb96be05f8c98ca4a87a384200ae" exitCode=0
Jul 29 20:46:52.132869 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:52.132708    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrct" event={"ID":"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1","Type":"ContainerDied","Data":"0903ec85a81a2ae25782eca65d6458568abbcb96be05f8c98ca4a87a384200ae"}
Jul 29 20:46:52.136416 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:52.136387    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt" event={"ID":"562c3be5-ccad-4fbb-beaa-2403134cf790","Type":"ContainerStarted","Data":"f01b3bd6caf841820ab6a3b0820a36901b1931349ced3ce3afa772b2169c3bff"}
Jul 29 20:46:52.136755 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:52.136681    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:52.136755 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:52.136747    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:52.136881 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:52.136797    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:52.152048 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:52.152024    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:52.152201 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:52.152139    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:46:52.176647 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:52.176597    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt" podStartSLOduration=8.259694018 podStartE2EDuration="28.176581511s" totalImagesPullingTime="19.916887493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:52.17624968 +0000 UTC m=+27.772541385" watchObservedRunningTime="2026-07-29 20:46:52.176581511 +0000 UTC m=+27.772873195"
Jul 29 20:46:53.227200 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:53.227011    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hdvwv"]
Jul 29 20:46:53.227742 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:53.227327    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:53.227742 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:53.227426    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:53.229925 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:53.229898    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rzcl7"]
Jul 29 20:46:53.230091 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:53.230012    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:53.230209 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:53.230100    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:53.230392 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:53.230359    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v5zlw"]
Jul 29 20:46:53.230476 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:53.230440    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:53.230542 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:53.230512    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:54.143239 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:54.143205    2584 generic.go:356] "Generic (PLEG): container finished" podID="c4bb2f8a-22d1-4fd5-9722-81d47ada14f1" containerID="7c0c7c362f743bdbf2299e012620d4368c17da00e19b580c795754c80fa331e3" exitCode=0
Jul 29 20:46:54.143416 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:54.143260    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrct" event={"ID":"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1","Type":"ContainerDied","Data":"7c0c7c362f743bdbf2299e012620d4368c17da00e19b580c795754c80fa331e3"}
Jul 29 20:46:54.861251 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:54.861215    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:54.861702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:54.861300    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:54.861702 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:54.861330    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:54.861702 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:54.861355    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:54.861702 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:54.861403    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:54.861702 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:54.861448    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:56.151113 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:56.151030    2584 generic.go:356] "Generic (PLEG): container finished" podID="c4bb2f8a-22d1-4fd5-9722-81d47ada14f1" containerID="9399a7716ac146c1fe5cc1f5e02a6c928547f097b65aef6837cf650b5f8c723e" exitCode=0
Jul 29 20:46:56.151113 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:56.151089    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrct" event={"ID":"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1","Type":"ContainerDied","Data":"9399a7716ac146c1fe5cc1f5e02a6c928547f097b65aef6837cf650b5f8c723e"}
Jul 29 20:46:56.859781 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:56.859743    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:56.859969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:56.859789    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:56.859969 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:56.859749    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:56.859969 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:56.859877    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:56.860161 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:56.859975    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:56.860161 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:56.860080    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:57.482759 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:57.482724    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:57.483191 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:57.482909    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:57.483191 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:57.482988    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret podName:eeebe086-c70d-4f85-afbb-5e159fe76372 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:29.482967919 +0000 UTC m=+65.079259584 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret") pod "global-pull-secret-syncer-rzcl7" (UID: "eeebe086-c70d-4f85-afbb-5e159fe76372") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:46:57.584025 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:57.583948    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs\") pod \"network-metrics-daemon-hdvwv\" (UID: \"f4edb7a0-def8-488e-a6ea-eca10d6f1452\") " pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:57.584205 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:57.584110    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:57.584205 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:57.584199    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs podName:f4edb7a0-def8-488e-a6ea-eca10d6f1452 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:29.584177448 +0000 UTC m=+65.180469128 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs") pod "network-metrics-daemon-hdvwv" (UID: "f4edb7a0-def8-488e-a6ea-eca10d6f1452") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:46:57.684689 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:57.684654    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwm82\" (UniqueName: \"kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82\") pod \"network-check-target-v5zlw\" (UID: \"c9d95b24-a79a-4f80-ad4c-34cde58ecf3d\") " pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:57.684863 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:57.684823    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:46:57.684863 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:57.684842    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:46:57.684863 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:57.684852    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lwm82 for pod openshift-network-diagnostics/network-check-target-v5zlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:57.685023 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:57.684905    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82 podName:c9d95b24-a79a-4f80-ad4c-34cde58ecf3d nodeName:}" failed. No retries permitted until 2026-07-29 20:47:29.684887206 +0000 UTC m=+65.281178882 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lwm82" (UniqueName: "kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82") pod "network-check-target-v5zlw" (UID: "c9d95b24-a79a-4f80-ad4c-34cde58ecf3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:46:58.859989 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:58.859949    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:46:58.860470 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:58.859949    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:46:58.860470 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:58.860082    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zlw" podUID="c9d95b24-a79a-4f80-ad4c-34cde58ecf3d"
Jul 29 20:46:58.860470 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:58.859956    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:46:58.860470 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:58.860179    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzcl7" podUID="eeebe086-c70d-4f85-afbb-5e159fe76372"
Jul 29 20:46:58.860470 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:46:58.860243    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdvwv" podUID="f4edb7a0-def8-488e-a6ea-eca10d6f1452"
Jul 29 20:46:59.728168 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.728125    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-194.ec2.internal" event="NodeReady"
Jul 29 20:46:59.728335 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.728297    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 20:46:59.779666 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.779627    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qnhjk"]
Jul 29 20:46:59.784090 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.784063    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qnhjk"
Jul 29 20:46:59.787722 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.787249    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 20:46:59.787722 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.787667    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cc48x\""
Jul 29 20:46:59.788160 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.788078    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 20:46:59.788379 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.788360    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 20:46:59.788461 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.788400    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d272j"]
Jul 29 20:46:59.791522 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.791502    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-668f4f769b-2kqfm"]
Jul 29 20:46:59.791631 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.791614    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d272j"
Jul 29 20:46:59.794091 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.794069    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 20:46:59.794211 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.794107    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-79k6t\""
Jul 29 20:46:59.794280 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.794216    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 20:46:59.794678 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.794658    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:46:59.799230 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.799211    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wf87m\""
Jul 29 20:46:59.799543 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.799521    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 20:46:59.799543 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.799540    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 20:46:59.800189 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.800173    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 20:46:59.811295 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.811275    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 20:46:59.827225 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.825973    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qnhjk"]
Jul 29 20:46:59.827225 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.826895    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-668f4f769b-2kqfm"]
Jul 29 20:46:59.827707 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.827680    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d272j"]
Jul 29 20:46:59.899273 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899235    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:46:59.899700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899289    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-certificates\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:46:59.899700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899321    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fb63f73-49a4-435a-b4c3-896c4d83c966-ca-trust-extracted\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:46:59.899700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899368    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53634e55-d5a1-44f7-baff-f9183c1fc7bd-config-volume\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:46:59.899700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899394    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53634e55-d5a1-44f7-baff-f9183c1fc7bd-tmp-dir\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:46:59.899700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899425    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j74d\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-kube-api-access-6j74d\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:46:59.899700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899455    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c357a9f-b01a-41c9-b36d-cdd506c9b60d-cert\") pod \"ingress-canary-qnhjk\" (UID: \"5c357a9f-b01a-41c9-b36d-cdd506c9b60d\") " pod="openshift-ingress-canary/ingress-canary-qnhjk"
Jul 29 20:46:59.899700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899479    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngbpm\" (UniqueName: \"kubernetes.io/projected/5c357a9f-b01a-41c9-b36d-cdd506c9b60d-kube-api-access-ngbpm\") pod \"ingress-canary-qnhjk\" (UID: \"5c357a9f-b01a-41c9-b36d-cdd506c9b60d\") " pod="openshift-ingress-canary/ingress-canary-qnhjk"
Jul 29 20:46:59.899700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899508    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:46:59.899700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899567    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fb63f73-49a4-435a-b4c3-896c4d83c966-image-registry-private-configuration\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:46:59.899700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899615    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fb63f73-49a4-435a-b4c3-896c4d83c966-trusted-ca\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:46:59.899700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899635    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fb63f73-49a4-435a-b4c3-896c4d83c966-installation-pull-secrets\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:46:59.899700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899680    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-bound-sa-token\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:46:59.900169 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:46:59.899762    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww9sm\" (UniqueName: \"kubernetes.io/projected/53634e55-d5a1-44f7-baff-f9183c1fc7bd-kube-api-access-ww9sm\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:00.000566 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.000488    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fb63f73-49a4-435a-b4c3-896c4d83c966-trusted-ca\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.000566 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.000525    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fb63f73-49a4-435a-b4c3-896c4d83c966-installation-pull-secrets\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.000566 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.000556    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-bound-sa-token\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.000848 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.000575    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww9sm\" (UniqueName: \"kubernetes.io/projected/53634e55-d5a1-44f7-baff-f9183c1fc7bd-kube-api-access-ww9sm\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:00.000848 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.000607    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:00.000848 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.000628    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-certificates\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.000848 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.000647    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fb63f73-49a4-435a-b4c3-896c4d83c966-ca-trust-extracted\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.000848 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.000699    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53634e55-d5a1-44f7-baff-f9183c1fc7bd-config-volume\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:00.000848 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.000728    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53634e55-d5a1-44f7-baff-f9183c1fc7bd-tmp-dir\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:00.000848 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:00.000750    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:47:00.000848 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.000759    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j74d\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-kube-api-access-6j74d\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.000848 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:00.000825    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls podName:53634e55-d5a1-44f7-baff-f9183c1fc7bd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:00.500800475 +0000 UTC m=+36.097092152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls") pod "dns-default-d272j" (UID: "53634e55-d5a1-44f7-baff-f9183c1fc7bd") : secret "dns-default-metrics-tls" not found
Jul 29 20:47:00.001311 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.000968    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c357a9f-b01a-41c9-b36d-cdd506c9b60d-cert\") pod \"ingress-canary-qnhjk\" (UID: \"5c357a9f-b01a-41c9-b36d-cdd506c9b60d\") " pod="openshift-ingress-canary/ingress-canary-qnhjk"
Jul 29 20:47:00.001311 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.001047    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngbpm\" (UniqueName: \"kubernetes.io/projected/5c357a9f-b01a-41c9-b36d-cdd506c9b60d-kube-api-access-ngbpm\") pod \"ingress-canary-qnhjk\" (UID: \"5c357a9f-b01a-41c9-b36d-cdd506c9b60d\") " pod="openshift-ingress-canary/ingress-canary-qnhjk"
Jul 29 20:47:00.001311 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.001097    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.001311 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.001157    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fb63f73-49a4-435a-b4c3-896c4d83c966-image-registry-private-configuration\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.001525 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.001494    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53634e55-d5a1-44f7-baff-f9183c1fc7bd-config-volume\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:00.001572 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:00.001502    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:47:00.001572 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:00.001554    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668f4f769b-2kqfm: secret "image-registry-tls" not found
Jul 29 20:47:00.001647 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:00.001626    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls podName:2fb63f73-49a4-435a-b4c3-896c4d83c966 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:00.501607764 +0000 UTC m=+36.097899443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls") pod "image-registry-668f4f769b-2kqfm" (UID: "2fb63f73-49a4-435a-b4c3-896c4d83c966") : secret "image-registry-tls" not found
Jul 29 20:47:00.001736 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.001703    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-certificates\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.001785 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.001762    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53634e55-d5a1-44f7-baff-f9183c1fc7bd-tmp-dir\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:00.001927 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.001907    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fb63f73-49a4-435a-b4c3-896c4d83c966-ca-trust-extracted\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.002113 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.002083    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fb63f73-49a4-435a-b4c3-896c4d83c966-trusted-ca\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.004990 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.004965    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fb63f73-49a4-435a-b4c3-896c4d83c966-image-registry-private-configuration\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.004990 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.004979    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c357a9f-b01a-41c9-b36d-cdd506c9b60d-cert\") pod \"ingress-canary-qnhjk\" (UID: \"5c357a9f-b01a-41c9-b36d-cdd506c9b60d\") " pod="openshift-ingress-canary/ingress-canary-qnhjk"
Jul 29 20:47:00.005164 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.004965    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fb63f73-49a4-435a-b4c3-896c4d83c966-installation-pull-secrets\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.009426 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.009381    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww9sm\" (UniqueName: \"kubernetes.io/projected/53634e55-d5a1-44f7-baff-f9183c1fc7bd-kube-api-access-ww9sm\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:00.009657 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.009640    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-bound-sa-token\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.009731 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.009693    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngbpm\" (UniqueName: \"kubernetes.io/projected/5c357a9f-b01a-41c9-b36d-cdd506c9b60d-kube-api-access-ngbpm\") pod \"ingress-canary-qnhjk\" (UID: \"5c357a9f-b01a-41c9-b36d-cdd506c9b60d\") " pod="openshift-ingress-canary/ingress-canary-qnhjk"
Jul 29 20:47:00.010099 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.010077    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j74d\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-kube-api-access-6j74d\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.095878 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.095844    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qnhjk"
Jul 29 20:47:00.240278 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.240249    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qnhjk"]
Jul 29 20:47:00.244501 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:47:00.244472    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c357a9f_b01a_41c9_b36d_cdd506c9b60d.slice/crio-139669aec898c121b8a6da94b81ff2a923e3aa14f6a1399a2b9cc144c92c5c47 WatchSource:0}: Error finding container 139669aec898c121b8a6da94b81ff2a923e3aa14f6a1399a2b9cc144c92c5c47: Status 404 returned error can't find the container with id 139669aec898c121b8a6da94b81ff2a923e3aa14f6a1399a2b9cc144c92c5c47
Jul 29 20:47:00.506039 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.505937    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:00.506039 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.506024    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:00.506241 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:00.506099    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:47:00.506241 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:00.506113    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:47:00.506241 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:00.506117    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668f4f769b-2kqfm: secret "image-registry-tls" not found
Jul 29 20:47:00.506241 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:00.506184    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls podName:53634e55-d5a1-44f7-baff-f9183c1fc7bd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:01.506166893 +0000 UTC m=+37.102458576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls") pod "dns-default-d272j" (UID: "53634e55-d5a1-44f7-baff-f9183c1fc7bd") : secret "dns-default-metrics-tls" not found
Jul 29 20:47:00.506241 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:00.506203    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls podName:2fb63f73-49a4-435a-b4c3-896c4d83c966 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:01.506189572 +0000 UTC m=+37.102481255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls") pod "image-registry-668f4f769b-2kqfm" (UID: "2fb63f73-49a4-435a-b4c3-896c4d83c966") : secret "image-registry-tls" not found
Jul 29 20:47:00.860091 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.860050    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:47:00.860291 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.860051    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:47:00.860291 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.860050    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:47:00.862440 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.862414    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 20:47:00.862570 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.862452    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zcmz5\""
Jul 29 20:47:00.862570 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.862461    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 20:47:00.862789 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.862772    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 20:47:00.863086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.863066    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lzlv4\""
Jul 29 20:47:00.863201 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:00.863091    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 20:47:01.169449 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:01.169173    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qnhjk" event={"ID":"5c357a9f-b01a-41c9-b36d-cdd506c9b60d","Type":"ContainerStarted","Data":"139669aec898c121b8a6da94b81ff2a923e3aa14f6a1399a2b9cc144c92c5c47"}
Jul 29 20:47:01.172133 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:01.172101    2584 generic.go:356] "Generic (PLEG): container finished" podID="c4bb2f8a-22d1-4fd5-9722-81d47ada14f1" containerID="31de028072b01b34aec5482e0e01da939df72827759faaa0eb24f0ce36685e65" exitCode=0
Jul 29 20:47:01.172250 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:01.172186    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrct" event={"ID":"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1","Type":"ContainerDied","Data":"31de028072b01b34aec5482e0e01da939df72827759faaa0eb24f0ce36685e65"}
Jul 29 20:47:01.516564 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:01.516446    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:01.516564 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:01.516549    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:01.516877 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:01.516605    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:47:01.516877 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:01.516678    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:47:01.516877 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:01.516688    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls podName:53634e55-d5a1-44f7-baff-f9183c1fc7bd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:03.516664987 +0000 UTC m=+39.112956668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls") pod "dns-default-d272j" (UID: "53634e55-d5a1-44f7-baff-f9183c1fc7bd") : secret "dns-default-metrics-tls" not found
Jul 29 20:47:01.516877 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:01.516694    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668f4f769b-2kqfm: secret "image-registry-tls" not found
Jul 29 20:47:01.516877 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:01.516735    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls podName:2fb63f73-49a4-435a-b4c3-896c4d83c966 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:03.516725148 +0000 UTC m=+39.113016835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls") pod "image-registry-668f4f769b-2kqfm" (UID: "2fb63f73-49a4-435a-b4c3-896c4d83c966") : secret "image-registry-tls" not found
Jul 29 20:47:02.177940 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:02.177904    2584 generic.go:356] "Generic (PLEG): container finished" podID="c4bb2f8a-22d1-4fd5-9722-81d47ada14f1" containerID="1e85a9e233d1926315046c970166160d4ea6c8179b811aff70b0723229629880" exitCode=0
Jul 29 20:47:02.178402 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:02.177978    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrct" event={"ID":"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1","Type":"ContainerDied","Data":"1e85a9e233d1926315046c970166160d4ea6c8179b811aff70b0723229629880"}
Jul 29 20:47:03.182226 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:03.182185    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qnhjk" event={"ID":"5c357a9f-b01a-41c9-b36d-cdd506c9b60d","Type":"ContainerStarted","Data":"0439d8bbf193538c2707b1ad8284ddf17d1e5d9114ff335ddaa0bb98e3db16a8"}
Jul 29 20:47:03.185514 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:03.185480    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrct" event={"ID":"c4bb2f8a-22d1-4fd5-9722-81d47ada14f1","Type":"ContainerStarted","Data":"c7afdb40086f4bd349351647431b02ed907fcce3db0f4ab3204104c558b6f2f7"}
Jul 29 20:47:03.197454 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:03.197395    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qnhjk" podStartSLOduration=2.054975731 podStartE2EDuration="4.197375215s" totalImagesPullingTime="2.142399484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:03.196383 +0000 UTC m=+38.792674684" watchObservedRunningTime="2026-07-29 20:47:03.197375215 +0000 UTC m=+38.793666901"
Jul 29 20:47:03.216301 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:03.216248    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cmrct" podStartSLOduration=9.298243051 podStartE2EDuration="39.216229904s" totalImagesPullingTime="29.917986853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:03.215619275 +0000 UTC m=+38.811910950" watchObservedRunningTime="2026-07-29 20:47:03.216229904 +0000 UTC m=+38.812521590"
Jul 29 20:47:03.535199 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:03.535083    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:03.535361 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:03.535251    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:47:03.535361 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:03.535272    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668f4f769b-2kqfm: secret "image-registry-tls" not found
Jul 29 20:47:03.535361 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:03.535256    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:03.535361 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:03.535326    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls podName:2fb63f73-49a4-435a-b4c3-896c4d83c966 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:07.535309809 +0000 UTC m=+43.131601492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls") pod "image-registry-668f4f769b-2kqfm" (UID: "2fb63f73-49a4-435a-b4c3-896c4d83c966") : secret "image-registry-tls" not found
Jul 29 20:47:03.535361 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:03.535335    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:47:03.535608 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:03.535387    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls podName:53634e55-d5a1-44f7-baff-f9183c1fc7bd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:07.53537239 +0000 UTC m=+43.131664072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls") pod "dns-default-d272j" (UID: "53634e55-d5a1-44f7-baff-f9183c1fc7bd") : secret "dns-default-metrics-tls" not found
Jul 29 20:47:07.566866 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:07.566824    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:07.567265 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:07.566887    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:07.567265 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:07.566981    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:47:07.567265 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:07.566991    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:47:07.567265 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:07.567010    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668f4f769b-2kqfm: secret "image-registry-tls" not found
Jul 29 20:47:07.567265 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:07.567036    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls podName:53634e55-d5a1-44f7-baff-f9183c1fc7bd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:15.567021617 +0000 UTC m=+51.163313279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls") pod "dns-default-d272j" (UID: "53634e55-d5a1-44f7-baff-f9183c1fc7bd") : secret "dns-default-metrics-tls" not found
Jul 29 20:47:07.567265 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:07.567058    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls podName:2fb63f73-49a4-435a-b4c3-896c4d83c966 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:15.567043537 +0000 UTC m=+51.163335199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls") pod "image-registry-668f4f769b-2kqfm" (UID: "2fb63f73-49a4-435a-b4c3-896c4d83c966") : secret "image-registry-tls" not found
Jul 29 20:47:08.710959 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.710921    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5xldl"]
Jul 29 20:47:08.749976 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.749934    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5xldl"]
Jul 29 20:47:08.750190 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.750080    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:08.751886 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.751863    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2kmm5\""
Jul 29 20:47:08.752579 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.752564    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 20:47:08.752670 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.752563    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 20:47:08.752670 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.752607    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 20:47:08.752670 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.752585    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 20:47:08.876629 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.876598    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw74g\" (UniqueName: \"kubernetes.io/projected/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-api-access-xw74g\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:08.876800 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.876640    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:08.876800 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.876717    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:08.876800 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.876750    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-crio-socket\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:08.876800 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.876768    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-data-volume\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:08.977196 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.977082    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw74g\" (UniqueName: \"kubernetes.io/projected/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-api-access-xw74g\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:08.977196 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.977127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:08.977440 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.977201    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:08.977440 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.977221    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-crio-socket\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:08.977440 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.977239    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-data-volume\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:08.977440 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:08.977324    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:47:08.977440 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:08.977342    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:47:08.977440 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:08.977403    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm podName:1ecfbe60-3617-43c2-bacf-f347a1e88b8c nodeName:}" failed. No retries permitted until 2026-07-29 20:47:09.477384838 +0000 UTC m=+45.073676522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xldl" (UID: "1ecfbe60-3617-43c2-bacf-f347a1e88b8c") : configmap "kube-rbac-proxy" not found
Jul 29 20:47:08.977440 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:08.977418    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls podName:1ecfbe60-3617-43c2-bacf-f347a1e88b8c nodeName:}" failed. No retries permitted until 2026-07-29 20:47:09.477411834 +0000 UTC m=+45.073703496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xldl" (UID: "1ecfbe60-3617-43c2-bacf-f347a1e88b8c") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:47:08.977440 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.977437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-crio-socket\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:08.977744 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.977601    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-data-volume\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:08.986323 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:08.986294    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw74g\" (UniqueName: \"kubernetes.io/projected/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-api-access-xw74g\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:09.481254 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:09.481208    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:09.481449 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:09.481304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:09.481449 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:09.481337    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:47:09.481449 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:09.481401    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm podName:1ecfbe60-3617-43c2-bacf-f347a1e88b8c nodeName:}" failed. No retries permitted until 2026-07-29 20:47:10.481382718 +0000 UTC m=+46.077674381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xldl" (UID: "1ecfbe60-3617-43c2-bacf-f347a1e88b8c") : configmap "kube-rbac-proxy" not found
Jul 29 20:47:09.481449 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:09.481439    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:47:09.481591 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:09.481490    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls podName:1ecfbe60-3617-43c2-bacf-f347a1e88b8c nodeName:}" failed. No retries permitted until 2026-07-29 20:47:10.481478495 +0000 UTC m=+46.077770159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xldl" (UID: "1ecfbe60-3617-43c2-bacf-f347a1e88b8c") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:47:10.488978 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:10.488937    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:10.489385 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:10.489012    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:10.489385 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:10.489093    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:47:10.489385 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:10.489095    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:47:10.489385 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:10.489165    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm podName:1ecfbe60-3617-43c2-bacf-f347a1e88b8c nodeName:}" failed. No retries permitted until 2026-07-29 20:47:12.489132553 +0000 UTC m=+48.085424228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xldl" (UID: "1ecfbe60-3617-43c2-bacf-f347a1e88b8c") : configmap "kube-rbac-proxy" not found
Jul 29 20:47:10.489385 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:10.489182    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls podName:1ecfbe60-3617-43c2-bacf-f347a1e88b8c nodeName:}" failed. No retries permitted until 2026-07-29 20:47:12.489173784 +0000 UTC m=+48.085465446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xldl" (UID: "1ecfbe60-3617-43c2-bacf-f347a1e88b8c") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:47:10.504678 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:10.504652    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2z8d_4b75e3fd-cc2d-4da4-9026-c8b6302dae66/dns-node-resolver/0.log"
Jul 29 20:47:11.503497 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:11.503470    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-562n4_042d3857-ba4b-4d0d-b9da-3ad4d6003cd3/node-ca/0.log"
Jul 29 20:47:12.303690 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:12.303659    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qnhjk_5c357a9f-b01a-41c9-b36d-cdd506c9b60d/serve-healthcheck-canary/0.log"
Jul 29 20:47:12.504889 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:12.504849    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:12.505355 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:12.504918    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:12.505355 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:12.504950    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:47:12.505355 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:12.505047    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm podName:1ecfbe60-3617-43c2-bacf-f347a1e88b8c nodeName:}" failed. No retries permitted until 2026-07-29 20:47:16.505024288 +0000 UTC m=+52.101315964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xldl" (UID: "1ecfbe60-3617-43c2-bacf-f347a1e88b8c") : configmap "kube-rbac-proxy" not found
Jul 29 20:47:12.505355 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:12.505076    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:47:12.505355 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:12.505131    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls podName:1ecfbe60-3617-43c2-bacf-f347a1e88b8c nodeName:}" failed. No retries permitted until 2026-07-29 20:47:16.50511845 +0000 UTC m=+52.101410137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xldl" (UID: "1ecfbe60-3617-43c2-bacf-f347a1e88b8c") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:47:15.625677 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:15.625643    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls\") pod \"image-registry-668f4f769b-2kqfm\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") " pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:15.626056 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:15.625710    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:15.626056 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:15.625800    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:47:15.626056 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:15.625825    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668f4f769b-2kqfm: secret "image-registry-tls" not found
Jul 29 20:47:15.626056 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:15.625877    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls podName:2fb63f73-49a4-435a-b4c3-896c4d83c966 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:31.625862223 +0000 UTC m=+67.222153885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls") pod "image-registry-668f4f769b-2kqfm" (UID: "2fb63f73-49a4-435a-b4c3-896c4d83c966") : secret "image-registry-tls" not found
Jul 29 20:47:15.626056 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:15.625797    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:47:15.626056 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:15.625932    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls podName:53634e55-d5a1-44f7-baff-f9183c1fc7bd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:31.625920446 +0000 UTC m=+67.222212129 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls") pod "dns-default-d272j" (UID: "53634e55-d5a1-44f7-baff-f9183c1fc7bd") : secret "dns-default-metrics-tls" not found
Jul 29 20:47:16.533108 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:16.533063    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:16.533332 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:16.533167    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:16.533332 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:16.533292    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:47:16.533454 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:16.533361    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls podName:1ecfbe60-3617-43c2-bacf-f347a1e88b8c nodeName:}" failed. No retries permitted until 2026-07-29 20:47:24.533343478 +0000 UTC m=+60.129635156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xldl" (UID: "1ecfbe60-3617-43c2-bacf-f347a1e88b8c") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:47:16.533639 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:16.533618    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:24.156069 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:24.156040    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lvfwt"
Jul 29 20:47:24.597387 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:24.597360    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:24.599665 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:24.599637    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ecfbe60-3617-43c2-bacf-f347a1e88b8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xldl\" (UID: \"1ecfbe60-3617-43c2-bacf-f347a1e88b8c\") " pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:24.659849 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:24.659811    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5xldl"
Jul 29 20:47:24.811010 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:24.810980    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5xldl"]
Jul 29 20:47:24.814093 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:47:24.814065    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecfbe60_3617_43c2_bacf_f347a1e88b8c.slice/crio-a16c0194710ba185cba8375dcd3802b4ab7d375d14c8625101cabe7ed99d1157 WatchSource:0}: Error finding container a16c0194710ba185cba8375dcd3802b4ab7d375d14c8625101cabe7ed99d1157: Status 404 returned error can't find the container with id a16c0194710ba185cba8375dcd3802b4ab7d375d14c8625101cabe7ed99d1157
Jul 29 20:47:25.236199 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:25.236160    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xldl" event={"ID":"1ecfbe60-3617-43c2-bacf-f347a1e88b8c","Type":"ContainerStarted","Data":"a16c0194710ba185cba8375dcd3802b4ab7d375d14c8625101cabe7ed99d1157"}
Jul 29 20:47:29.537375 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.537337    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:47:29.539570 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.539549    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 20:47:29.549946 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.549920    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eeebe086-c70d-4f85-afbb-5e159fe76372-original-pull-secret\") pod \"global-pull-secret-syncer-rzcl7\" (UID: \"eeebe086-c70d-4f85-afbb-5e159fe76372\") " pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:47:29.637697 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.637663    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs\") pod \"network-metrics-daemon-hdvwv\" (UID: \"f4edb7a0-def8-488e-a6ea-eca10d6f1452\") " pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:47:29.639408 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.639389    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 20:47:29.650001 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.649981    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4edb7a0-def8-488e-a6ea-eca10d6f1452-metrics-certs\") pod \"network-metrics-daemon-hdvwv\" (UID: \"f4edb7a0-def8-488e-a6ea-eca10d6f1452\") " pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:47:29.680041 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.680010    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lzlv4\""
Jul 29 20:47:29.684536 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.684515    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzcl7"
Jul 29 20:47:29.688340 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.688310    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdvwv"
Jul 29 20:47:29.738915 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.738874    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwm82\" (UniqueName: \"kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82\") pod \"network-check-target-v5zlw\" (UID: \"c9d95b24-a79a-4f80-ad4c-34cde58ecf3d\") " pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:47:29.741081 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.741022    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 20:47:29.752275 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.751705    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 20:47:29.762824 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.762798    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwm82\" (UniqueName: \"kubernetes.io/projected/c9d95b24-a79a-4f80-ad4c-34cde58ecf3d-kube-api-access-lwm82\") pod \"network-check-target-v5zlw\" (UID: \"c9d95b24-a79a-4f80-ad4c-34cde58ecf3d\") " pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:47:29.810520 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.810447    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rzcl7"]
Jul 29 20:47:29.814731 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:47:29.814706    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeebe086_c70d_4f85_afbb_5e159fe76372.slice/crio-cb5217644fbe87063a4ebdbb549cf981e8fc1668c3c84a69a3ea405837543f98 WatchSource:0}: Error finding container cb5217644fbe87063a4ebdbb549cf981e8fc1668c3c84a69a3ea405837543f98: Status 404 returned error can't find the container with id cb5217644fbe87063a4ebdbb549cf981e8fc1668c3c84a69a3ea405837543f98
Jul 29 20:47:29.826220 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.826198    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hdvwv"]
Jul 29 20:47:29.828803 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:47:29.828782    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4edb7a0_def8_488e_a6ea_eca10d6f1452.slice/crio-3351bf6c3dc805ff9c4850e8ec0a304992ef3b73210d9f0949d49fbcbcbe0296 WatchSource:0}: Error finding container 3351bf6c3dc805ff9c4850e8ec0a304992ef3b73210d9f0949d49fbcbcbe0296: Status 404 returned error can't find the container with id 3351bf6c3dc805ff9c4850e8ec0a304992ef3b73210d9f0949d49fbcbcbe0296
Jul 29 20:47:29.983396 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.983365    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zcmz5\""
Jul 29 20:47:29.992859 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:29.992837    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:47:30.032599 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.032562    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"]
Jul 29 20:47:30.034830 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.034811    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"]
Jul 29 20:47:30.034961 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.034940    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"
Jul 29 20:47:30.036988 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.036874    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 20:47:30.039000 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.037287    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.039000 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.037304    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 20:47:30.039000 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.038261    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 29 20:47:30.039000 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.038649    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 20:47:30.039439 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.039254    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 29 20:47:30.039439 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.039286    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 29 20:47:30.041647 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.041607    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 29 20:47:30.041798 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.041611    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 29 20:47:30.047193 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.047040    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"]
Jul 29 20:47:30.050696 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.050647    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"]
Jul 29 20:47:30.099811 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.099781    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-668f4f769b-2kqfm"]
Jul 29 20:47:30.100009 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:30.099989    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-668f4f769b-2kqfm" podUID="2fb63f73-49a4-435a-b4c3-896c4d83c966"
Jul 29 20:47:30.127077 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.127046    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v5zlw"]
Jul 29 20:47:30.130088 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:47:30.130061    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d95b24_a79a_4f80_ad4c_34cde58ecf3d.slice/crio-bf6e4e37acf192aa4e5bb640ccb95d36a8b4369012db0d78b72a7e1164a2fcf5 WatchSource:0}: Error finding container bf6e4e37acf192aa4e5bb640ccb95d36a8b4369012db0d78b72a7e1164a2fcf5: Status 404 returned error can't find the container with id bf6e4e37acf192aa4e5bb640ccb95d36a8b4369012db0d78b72a7e1164a2fcf5
Jul 29 20:47:30.141242 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.141207    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/89c8ae13-f6d3-4f6a-a772-731a9c721370-ca\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.141346 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.141249    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmh44\" (UniqueName: \"kubernetes.io/projected/89c8ae13-f6d3-4f6a-a772-731a9c721370-kube-api-access-fmh44\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.141346 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.141289    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq6hw\" (UniqueName: \"kubernetes.io/projected/1aad9461-3286-4094-be39-822fea25a047-kube-api-access-zq6hw\") pod \"klusterlet-addon-workmgr-657f4d49cd-v2dzl\" (UID: \"1aad9461-3286-4094-be39-822fea25a047\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"
Jul 29 20:47:30.141346 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.141308    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1aad9461-3286-4094-be39-822fea25a047-tmp\") pod \"klusterlet-addon-workmgr-657f4d49cd-v2dzl\" (UID: \"1aad9461-3286-4094-be39-822fea25a047\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"
Jul 29 20:47:30.141346 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.141334    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/89c8ae13-f6d3-4f6a-a772-731a9c721370-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.141545 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.141415    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/89c8ae13-f6d3-4f6a-a772-731a9c721370-hub\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.141545 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.141455    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1aad9461-3286-4094-be39-822fea25a047-klusterlet-config\") pod \"klusterlet-addon-workmgr-657f4d49cd-v2dzl\" (UID: \"1aad9461-3286-4094-be39-822fea25a047\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"
Jul 29 20:47:30.141545 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.141483    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89c8ae13-f6d3-4f6a-a772-731a9c721370-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.141545 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.141507    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c8ae13-f6d3-4f6a-a772-731a9c721370-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.242026 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.241986    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq6hw\" (UniqueName: \"kubernetes.io/projected/1aad9461-3286-4094-be39-822fea25a047-kube-api-access-zq6hw\") pod \"klusterlet-addon-workmgr-657f4d49cd-v2dzl\" (UID: \"1aad9461-3286-4094-be39-822fea25a047\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"
Jul 29 20:47:30.242026 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.242031    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1aad9461-3286-4094-be39-822fea25a047-tmp\") pod \"klusterlet-addon-workmgr-657f4d49cd-v2dzl\" (UID: \"1aad9461-3286-4094-be39-822fea25a047\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"
Jul 29 20:47:30.242284 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.242069    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/89c8ae13-f6d3-4f6a-a772-731a9c721370-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.242284 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.242090    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/89c8ae13-f6d3-4f6a-a772-731a9c721370-hub\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.242284 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.242109    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1aad9461-3286-4094-be39-822fea25a047-klusterlet-config\") pod \"klusterlet-addon-workmgr-657f4d49cd-v2dzl\" (UID: \"1aad9461-3286-4094-be39-822fea25a047\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"
Jul 29 20:47:30.242284 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.242125    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89c8ae13-f6d3-4f6a-a772-731a9c721370-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.242503 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.242377    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c8ae13-f6d3-4f6a-a772-731a9c721370-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.242503 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.242479    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/89c8ae13-f6d3-4f6a-a772-731a9c721370-ca\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.242614 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.242512    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmh44\" (UniqueName: \"kubernetes.io/projected/89c8ae13-f6d3-4f6a-a772-731a9c721370-kube-api-access-fmh44\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.242614 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.242513    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1aad9461-3286-4094-be39-822fea25a047-tmp\") pod \"klusterlet-addon-workmgr-657f4d49cd-v2dzl\" (UID: \"1aad9461-3286-4094-be39-822fea25a047\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"
Jul 29 20:47:30.243270 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.243239    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c8ae13-f6d3-4f6a-a772-731a9c721370-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.244867 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.244841    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/89c8ae13-f6d3-4f6a-a772-731a9c721370-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.244963 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.244895    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/89c8ae13-f6d3-4f6a-a772-731a9c721370-ca\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.244963 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.244953    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/89c8ae13-f6d3-4f6a-a772-731a9c721370-hub\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.245267 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.245215    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89c8ae13-f6d3-4f6a-a772-731a9c721370-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.245399 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.245380    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1aad9461-3286-4094-be39-822fea25a047-klusterlet-config\") pod \"klusterlet-addon-workmgr-657f4d49cd-v2dzl\" (UID: \"1aad9461-3286-4094-be39-822fea25a047\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"
Jul 29 20:47:30.249563 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.249519    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq6hw\" (UniqueName: \"kubernetes.io/projected/1aad9461-3286-4094-be39-822fea25a047-kube-api-access-zq6hw\") pod \"klusterlet-addon-workmgr-657f4d49cd-v2dzl\" (UID: \"1aad9461-3286-4094-be39-822fea25a047\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"
Jul 29 20:47:30.250094 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.250073    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmh44\" (UniqueName: \"kubernetes.io/projected/89c8ae13-f6d3-4f6a-a772-731a9c721370-kube-api-access-fmh44\") pod \"cluster-proxy-proxy-agent-85f45d4769-hh7cn\" (UID: \"89c8ae13-f6d3-4f6a-a772-731a9c721370\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.250423 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.250396    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xldl" event={"ID":"1ecfbe60-3617-43c2-bacf-f347a1e88b8c","Type":"ContainerStarted","Data":"9f7aaecdec216781757b86eb689269d4bfaa85b58d7292f543c378014d87a4e9"}
Jul 29 20:47:30.251387 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.251365    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v5zlw" event={"ID":"c9d95b24-a79a-4f80-ad4c-34cde58ecf3d","Type":"ContainerStarted","Data":"bf6e4e37acf192aa4e5bb640ccb95d36a8b4369012db0d78b72a7e1164a2fcf5"}
Jul 29 20:47:30.252346 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.252326    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hdvwv" event={"ID":"f4edb7a0-def8-488e-a6ea-eca10d6f1452","Type":"ContainerStarted","Data":"3351bf6c3dc805ff9c4850e8ec0a304992ef3b73210d9f0949d49fbcbcbe0296"}
Jul 29 20:47:30.253412 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.253392    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:30.253412 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.253406    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rzcl7" event={"ID":"eeebe086-c70d-4f85-afbb-5e159fe76372","Type":"ContainerStarted","Data":"cb5217644fbe87063a4ebdbb549cf981e8fc1668c3c84a69a3ea405837543f98"}
Jul 29 20:47:30.258663 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.258646    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:30.344114 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.343086    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fb63f73-49a4-435a-b4c3-896c4d83c966-installation-pull-secrets\") pod \"2fb63f73-49a4-435a-b4c3-896c4d83c966\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") "
Jul 29 20:47:30.344114 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.343129    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-bound-sa-token\") pod \"2fb63f73-49a4-435a-b4c3-896c4d83c966\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") "
Jul 29 20:47:30.344114 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.343199    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fb63f73-49a4-435a-b4c3-896c4d83c966-trusted-ca\") pod \"2fb63f73-49a4-435a-b4c3-896c4d83c966\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") "
Jul 29 20:47:30.344114 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.343237    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-certificates\") pod \"2fb63f73-49a4-435a-b4c3-896c4d83c966\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") "
Jul 29 20:47:30.344114 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.343276    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fb63f73-49a4-435a-b4c3-896c4d83c966-ca-trust-extracted\") pod \"2fb63f73-49a4-435a-b4c3-896c4d83c966\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") "
Jul 29 20:47:30.344114 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.343307    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j74d\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-kube-api-access-6j74d\") pod \"2fb63f73-49a4-435a-b4c3-896c4d83c966\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") "
Jul 29 20:47:30.344114 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.343334    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fb63f73-49a4-435a-b4c3-896c4d83c966-image-registry-private-configuration\") pod \"2fb63f73-49a4-435a-b4c3-896c4d83c966\" (UID: \"2fb63f73-49a4-435a-b4c3-896c4d83c966\") "
Jul 29 20:47:30.344609 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.344164    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb63f73-49a4-435a-b4c3-896c4d83c966-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2fb63f73-49a4-435a-b4c3-896c4d83c966" (UID: "2fb63f73-49a4-435a-b4c3-896c4d83c966"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:47:30.344609 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.344518    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb63f73-49a4-435a-b4c3-896c4d83c966-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2fb63f73-49a4-435a-b4c3-896c4d83c966" (UID: "2fb63f73-49a4-435a-b4c3-896c4d83c966"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:47:30.344730 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.344686    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2fb63f73-49a4-435a-b4c3-896c4d83c966" (UID: "2fb63f73-49a4-435a-b4c3-896c4d83c966"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:47:30.347306 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.347278    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb63f73-49a4-435a-b4c3-896c4d83c966-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2fb63f73-49a4-435a-b4c3-896c4d83c966" (UID: "2fb63f73-49a4-435a-b4c3-896c4d83c966"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:47:30.349337 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.349307    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2fb63f73-49a4-435a-b4c3-896c4d83c966" (UID: "2fb63f73-49a4-435a-b4c3-896c4d83c966"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:47:30.350437 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.350394    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"
Jul 29 20:47:30.351449 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.351380    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb63f73-49a4-435a-b4c3-896c4d83c966-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2fb63f73-49a4-435a-b4c3-896c4d83c966" (UID: "2fb63f73-49a4-435a-b4c3-896c4d83c966"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:47:30.351449 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.351427    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-kube-api-access-6j74d" (OuterVolumeSpecName: "kube-api-access-6j74d") pod "2fb63f73-49a4-435a-b4c3-896c4d83c966" (UID: "2fb63f73-49a4-435a-b4c3-896c4d83c966"). InnerVolumeSpecName "kube-api-access-6j74d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:47:30.366294 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.366025    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"
Jul 29 20:47:30.444109 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.444069    2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fb63f73-49a4-435a-b4c3-896c4d83c966-ca-trust-extracted\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:47:30.444109 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.444108    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j74d\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-kube-api-access-6j74d\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:47:30.444356 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.444123    2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fb63f73-49a4-435a-b4c3-896c4d83c966-image-registry-private-configuration\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:47:30.444356 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.444137    2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fb63f73-49a4-435a-b4c3-896c4d83c966-installation-pull-secrets\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:47:30.444356 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.444168    2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-bound-sa-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:47:30.444356 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.444181    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fb63f73-49a4-435a-b4c3-896c4d83c966-trusted-ca\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:47:30.444356 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.444193    2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-certificates\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:47:30.532798 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.532651    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"]
Jul 29 20:47:30.535294 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:47:30.535263    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aad9461_3286_4094_be39_822fea25a047.slice/crio-ad8118b057c72d1c8d8e7b3e035fb4cca2adec3d90f08299b528d2e9181f4eff WatchSource:0}: Error finding container ad8118b057c72d1c8d8e7b3e035fb4cca2adec3d90f08299b528d2e9181f4eff: Status 404 returned error can't find the container with id ad8118b057c72d1c8d8e7b3e035fb4cca2adec3d90f08299b528d2e9181f4eff
Jul 29 20:47:30.550106 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:30.550046    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn"]
Jul 29 20:47:30.553942 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:47:30.553893    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89c8ae13_f6d3_4f6a_a772_731a9c721370.slice/crio-beaf4831573af44a9f309b2b534f070b197f0f320ab6691f2d85b7c48511f438 WatchSource:0}: Error finding container beaf4831573af44a9f309b2b534f070b197f0f320ab6691f2d85b7c48511f438: Status 404 returned error can't find the container with id beaf4831573af44a9f309b2b534f070b197f0f320ab6691f2d85b7c48511f438
Jul 29 20:47:31.258753 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:31.258660    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl" event={"ID":"1aad9461-3286-4094-be39-822fea25a047","Type":"ContainerStarted","Data":"ad8118b057c72d1c8d8e7b3e035fb4cca2adec3d90f08299b528d2e9181f4eff"}
Jul 29 20:47:31.260018 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:31.259992    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668f4f769b-2kqfm"
Jul 29 20:47:31.260299 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:31.260251    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn" event={"ID":"89c8ae13-f6d3-4f6a-a772-731a9c721370","Type":"ContainerStarted","Data":"beaf4831573af44a9f309b2b534f070b197f0f320ab6691f2d85b7c48511f438"}
Jul 29 20:47:31.289510 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:31.289428    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-668f4f769b-2kqfm"]
Jul 29 20:47:31.294937 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:31.294883    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-668f4f769b-2kqfm"]
Jul 29 20:47:31.353425 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:31.353380    2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fb63f73-49a4-435a-b4c3-896c4d83c966-registry-tls\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:47:31.656562 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:31.656531    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:31.669219 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:31.669159    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53634e55-d5a1-44f7-baff-f9183c1fc7bd-metrics-tls\") pod \"dns-default-d272j\" (UID: \"53634e55-d5a1-44f7-baff-f9183c1fc7bd\") " pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:31.906117 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:31.906077    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-79k6t\""
Jul 29 20:47:31.915569 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:31.915527    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:32.132826 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:32.132627    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d272j"]
Jul 29 20:47:32.145212 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:47:32.145181    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53634e55_d5a1_44f7_baff_f9183c1fc7bd.slice/crio-5665129c97b0f746dd731826687d4b9b1ae2480512654162bfca1b82faa5195c WatchSource:0}: Error finding container 5665129c97b0f746dd731826687d4b9b1ae2480512654162bfca1b82faa5195c: Status 404 returned error can't find the container with id 5665129c97b0f746dd731826687d4b9b1ae2480512654162bfca1b82faa5195c
Jul 29 20:47:32.268102 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:32.267977    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d272j" event={"ID":"53634e55-d5a1-44f7-baff-f9183c1fc7bd","Type":"ContainerStarted","Data":"5665129c97b0f746dd731826687d4b9b1ae2480512654162bfca1b82faa5195c"}
Jul 29 20:47:32.276974 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:32.276172    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hdvwv" event={"ID":"f4edb7a0-def8-488e-a6ea-eca10d6f1452","Type":"ContainerStarted","Data":"aab4df1b3eaf968c6ee4b76e686cbea3a203020df2f9b6de908053cabe939670"}
Jul 29 20:47:32.276974 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:32.276236    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hdvwv" event={"ID":"f4edb7a0-def8-488e-a6ea-eca10d6f1452","Type":"ContainerStarted","Data":"2c74d29cdf9bd5c4442de686525befdd30a74994ea74485bbdd7daf27ac7a7cb"}
Jul 29 20:47:32.291276 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:32.290717    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hdvwv" podStartSLOduration=66.576535956 podStartE2EDuration="1m8.290696132s" totalImagesPullingTime="1.714160176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:32.28977874 +0000 UTC m=+67.886070449" watchObservedRunningTime="2026-07-29 20:47:32.290696132 +0000 UTC m=+67.886987817"
Jul 29 20:47:32.865775 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:32.865716    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fb63f73-49a4-435a-b4c3-896c4d83c966" path="/var/lib/kubelet/pods/2fb63f73-49a4-435a-b4c3-896c4d83c966/volumes"
Jul 29 20:47:41.314091 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.314018    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xldl" event={"ID":"1ecfbe60-3617-43c2-bacf-f347a1e88b8c","Type":"ContainerStarted","Data":"5c21ce2745d59711b21599a99c5dcb85df67a5cf7ea585b36a14ed986f374166"}
Jul 29 20:47:41.316198 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.316166    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn" event={"ID":"89c8ae13-f6d3-4f6a-a772-731a9c721370","Type":"ContainerStarted","Data":"0d7a1ff0ee084f9adaa57e63a0573073c0ecf5d52f6e7a3ad0e7ab3df2ff05b3"}
Jul 29 20:47:41.316302 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.316216    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn" event={"ID":"89c8ae13-f6d3-4f6a-a772-731a9c721370","Type":"ContainerStarted","Data":"bd80307fe80b3224e9e814df00eaeb544d225ea3caf3f327042a4a6c4d4e60a8"}
Jul 29 20:47:41.317741 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.317715    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v5zlw" event={"ID":"c9d95b24-a79a-4f80-ad4c-34cde58ecf3d","Type":"ContainerStarted","Data":"689b2c493ed5a59b705007fb88becafe9c6b67564fd8ffd43086d085551727ab"}
Jul 29 20:47:41.317846 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.317789    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:47:41.319013 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.318991    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rzcl7" event={"ID":"eeebe086-c70d-4f85-afbb-5e159fe76372","Type":"ContainerStarted","Data":"02d52057ddfeb5dca6e8822475e2036c383cf7d643a3db5eea9e44fdc929b52d"}
Jul 29 20:47:41.320952 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.320930    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl" event={"ID":"1aad9461-3286-4094-be39-822fea25a047","Type":"ContainerStarted","Data":"9da75e4f726e328e3c71ab9d652b9c1eb96a58b30b41b913e908074614ebcc31"}
Jul 29 20:47:41.321253 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.321079    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"
Jul 29 20:47:41.322984 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.322857    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl"
Jul 29 20:47:41.323355 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.323336    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d272j" event={"ID":"53634e55-d5a1-44f7-baff-f9183c1fc7bd","Type":"ContainerStarted","Data":"97198c6da0a9d2d6cdeae5f886de8c37ce10019324e04e7974ebf5fe33147678"}
Jul 29 20:47:41.323444 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.323379    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d272j" event={"ID":"53634e55-d5a1-44f7-baff-f9183c1fc7bd","Type":"ContainerStarted","Data":"cc1ef2f7a67261adce3aba57e2b8cb3762acefc49cd577ad48a5200fe36e7f2a"}
Jul 29 20:47:41.323444 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.323417    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:41.331727 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.331645    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v5zlw" podStartSLOduration=66.423360959 podStartE2EDuration="1m17.331626489s" totalImagesPullingTime="10.90826553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:41.330897703 +0000 UTC m=+76.927189405" watchObservedRunningTime="2026-07-29 20:47:41.331626489 +0000 UTC m=+76.927918174"
Jul 29 20:47:41.349619 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.345973    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657f4d49cd-v2dzl" podStartSLOduration=0.820998385 podStartE2EDuration="11.345954578s" totalImagesPullingTime="10.524956193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:41.344690559 +0000 UTC m=+76.940982243" watchObservedRunningTime="2026-07-29 20:47:41.345954578 +0000 UTC m=+76.942246262"
Jul 29 20:47:41.361191 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.361122    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rzcl7" podStartSLOduration=66.114957627 podStartE2EDuration="1m17.361101671s" totalImagesPullingTime="11.246144044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:41.360983339 +0000 UTC m=+76.957275035" watchObservedRunningTime="2026-07-29 20:47:41.361101671 +0000 UTC m=+76.957393335"
Jul 29 20:47:41.374503 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:41.374411    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d272j" podStartSLOduration=33.439107853 podStartE2EDuration="42.374395827s" totalImagesPullingTime="8.935287974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:41.373884248 +0000 UTC m=+76.970175932" watchObservedRunningTime="2026-07-29 20:47:41.374395827 +0000 UTC m=+76.970687512"
Jul 29 20:47:42.331696 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:42.331654    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn" event={"ID":"89c8ae13-f6d3-4f6a-a772-731a9c721370","Type":"ContainerStarted","Data":"bbe0304f2919afb18fe209ecb1617bffeb7b19ae9bad20629855fd8fbb992ce2"}
Jul 29 20:47:42.348078 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:42.348020    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f45d4769-hh7cn" podStartSLOduration=1.842785702 podStartE2EDuration="12.348000126s" totalImagesPullingTime="10.505214424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:42.347523993 +0000 UTC m=+77.943815701" watchObservedRunningTime="2026-07-29 20:47:42.348000126 +0000 UTC m=+77.944291812"
Jul 29 20:47:45.344685 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:45.344647    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xldl" event={"ID":"1ecfbe60-3617-43c2-bacf-f347a1e88b8c","Type":"ContainerStarted","Data":"1162c785c88176db3bc54cab68a5bd13fd1a56748c364e55f4753ba861a0dad1"}
Jul 29 20:47:45.359910 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:45.359863    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5xldl" podStartSLOduration=17.711602263 podStartE2EDuration="37.359849539s" totalImagesPullingTime="19.648247276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:45.359719681 +0000 UTC m=+80.956011391" watchObservedRunningTime="2026-07-29 20:47:45.359849539 +0000 UTC m=+80.956141223"
Jul 29 20:47:50.814872 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.814756    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"]
Jul 29 20:47:50.819185 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.819169    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:50.821264 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.821238    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 20:47:50.821264 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.821251    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 20:47:50.821416 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.821238    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 20:47:50.821588 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.821573    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 29 20:47:50.821654 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.821599    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-25wbr\""
Jul 29 20:47:50.821710 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.821603    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 29 20:47:50.826885 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.826866    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"]
Jul 29 20:47:50.832731 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.832711    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hnqdd"]
Jul 29 20:47:50.835635 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.835615    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.837382 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.837364    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 20:47:50.837464 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.837392    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xmffz\""
Jul 29 20:47:50.837624 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.837609    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 20:47:50.837662 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.837616    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 20:47:50.898739 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.898709    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-wtmp\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.898895 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.898738    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-textfile\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.898895 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.898769    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.898895 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.898845    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f85408-f650-4bb2-84c5-6943cefc4c5f-metrics-client-ca\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.898998 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.898904    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ms4q\" (UniqueName: \"kubernetes.io/projected/8c4999cf-05ac-4056-82ff-9917e356f0bb-kube-api-access-5ms4q\") pod \"openshift-state-metrics-b8f49f5c6-26pn4\" (UID: \"8c4999cf-05ac-4056-82ff-9917e356f0bb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:50.898998 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.898928    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1f85408-f650-4bb2-84c5-6943cefc4c5f-sys\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.898998 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.898950    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.898998 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.898971    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c4999cf-05ac-4056-82ff-9917e356f0bb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-26pn4\" (UID: \"8c4999cf-05ac-4056-82ff-9917e356f0bb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:50.898998 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.898993    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c4999cf-05ac-4056-82ff-9917e356f0bb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-26pn4\" (UID: \"8c4999cf-05ac-4056-82ff-9917e356f0bb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:50.899162 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.899018    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l92j\" (UniqueName: \"kubernetes.io/projected/f1f85408-f650-4bb2-84c5-6943cefc4c5f-kube-api-access-8l92j\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.899162 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.899037    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c4999cf-05ac-4056-82ff-9917e356f0bb-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-26pn4\" (UID: \"8c4999cf-05ac-4056-82ff-9917e356f0bb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:50.899162 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.899056    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1f85408-f650-4bb2-84c5-6943cefc4c5f-root\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.899162 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.899077    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-tls\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.999652 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999613    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1f85408-f650-4bb2-84c5-6943cefc4c5f-root\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.999652 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999654    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-tls\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.999901 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999682    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-wtmp\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.999901 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999706    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-textfile\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.999901 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999727    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.999901 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999738    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1f85408-f650-4bb2-84c5-6943cefc4c5f-root\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.999901 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999756    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f85408-f650-4bb2-84c5-6943cefc4c5f-metrics-client-ca\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.999901 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999819    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ms4q\" (UniqueName: \"kubernetes.io/projected/8c4999cf-05ac-4056-82ff-9917e356f0bb-kube-api-access-5ms4q\") pod \"openshift-state-metrics-b8f49f5c6-26pn4\" (UID: \"8c4999cf-05ac-4056-82ff-9917e356f0bb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:50.999901 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999849    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1f85408-f650-4bb2-84c5-6943cefc4c5f-sys\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.999901 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999887    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:50.999901 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:50.999892    2584 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 20:47:51.000364 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999919    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c4999cf-05ac-4056-82ff-9917e356f0bb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-26pn4\" (UID: \"8c4999cf-05ac-4056-82ff-9917e356f0bb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:51.000364 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999935    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1f85408-f650-4bb2-84c5-6943cefc4c5f-sys\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:51.000364 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999943    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c4999cf-05ac-4056-82ff-9917e356f0bb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-26pn4\" (UID: \"8c4999cf-05ac-4056-82ff-9917e356f0bb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:51.000364 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:50.999965    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-tls podName:f1f85408-f650-4bb2-84c5-6943cefc4c5f nodeName:}" failed. No retries permitted until 2026-07-29 20:47:51.499945 +0000 UTC m=+87.096236678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-tls") pod "node-exporter-hnqdd" (UID: "f1f85408-f650-4bb2-84c5-6943cefc4c5f") : secret "node-exporter-tls" not found
Jul 29 20:47:51.000364 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.000016    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l92j\" (UniqueName: \"kubernetes.io/projected/f1f85408-f650-4bb2-84c5-6943cefc4c5f-kube-api-access-8l92j\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:51.000364 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.000033    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-textfile\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:51.000364 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:50.999822    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-wtmp\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:51.000364 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.000051    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c4999cf-05ac-4056-82ff-9917e356f0bb-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-26pn4\" (UID: \"8c4999cf-05ac-4056-82ff-9917e356f0bb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:51.000364 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:51.000186    2584 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 29 20:47:51.000364 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:47:51.000249    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4999cf-05ac-4056-82ff-9917e356f0bb-openshift-state-metrics-tls podName:8c4999cf-05ac-4056-82ff-9917e356f0bb nodeName:}" failed. No retries permitted until 2026-07-29 20:47:51.500229287 +0000 UTC m=+87.096520983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8c4999cf-05ac-4056-82ff-9917e356f0bb-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-26pn4" (UID: "8c4999cf-05ac-4056-82ff-9917e356f0bb") : secret "openshift-state-metrics-tls" not found
Jul 29 20:47:51.000769 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.000418    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f85408-f650-4bb2-84c5-6943cefc4c5f-metrics-client-ca\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:51.000769 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.000446    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:51.000769 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.000629    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c4999cf-05ac-4056-82ff-9917e356f0bb-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-26pn4\" (UID: \"8c4999cf-05ac-4056-82ff-9917e356f0bb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:51.002351 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.002332    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:51.002427 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.002412    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c4999cf-05ac-4056-82ff-9917e356f0bb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-26pn4\" (UID: \"8c4999cf-05ac-4056-82ff-9917e356f0bb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:51.013911 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.013886    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l92j\" (UniqueName: \"kubernetes.io/projected/f1f85408-f650-4bb2-84c5-6943cefc4c5f-kube-api-access-8l92j\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:51.014070 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.014050    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ms4q\" (UniqueName: \"kubernetes.io/projected/8c4999cf-05ac-4056-82ff-9917e356f0bb-kube-api-access-5ms4q\") pod \"openshift-state-metrics-b8f49f5c6-26pn4\" (UID: \"8c4999cf-05ac-4056-82ff-9917e356f0bb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:51.334228 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.334195    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d272j"
Jul 29 20:47:51.505628 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.505593    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-tls\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:51.505836 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.505732    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c4999cf-05ac-4056-82ff-9917e356f0bb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-26pn4\" (UID: \"8c4999cf-05ac-4056-82ff-9917e356f0bb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:51.508035 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.508009    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1f85408-f650-4bb2-84c5-6943cefc4c5f-node-exporter-tls\") pod \"node-exporter-hnqdd\" (UID: \"f1f85408-f650-4bb2-84c5-6943cefc4c5f\") " pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:51.508136 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.508067    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c4999cf-05ac-4056-82ff-9917e356f0bb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-26pn4\" (UID: \"8c4999cf-05ac-4056-82ff-9917e356f0bb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:51.729161 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.729108    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"
Jul 29 20:47:51.744538 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.744505    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hnqdd"
Jul 29 20:47:51.754417 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:47:51.754393    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f85408_f650_4bb2_84c5_6943cefc4c5f.slice/crio-5bc6736b99887d538186f0950e901a348cd745b15861ec2c723d85c81a3fd36a WatchSource:0}: Error finding container 5bc6736b99887d538186f0950e901a348cd745b15861ec2c723d85c81a3fd36a: Status 404 returned error can't find the container with id 5bc6736b99887d538186f0950e901a348cd745b15861ec2c723d85c81a3fd36a
Jul 29 20:47:51.859348 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:51.859317    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4"]
Jul 29 20:47:51.863410 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:47:51.863380    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4999cf_05ac_4056_82ff_9917e356f0bb.slice/crio-cce1ef2c5e370c5acbc878524bfa5e0a220ee37c5d10f6e6a4ca70e08c5624df WatchSource:0}: Error finding container cce1ef2c5e370c5acbc878524bfa5e0a220ee37c5d10f6e6a4ca70e08c5624df: Status 404 returned error can't find the container with id cce1ef2c5e370c5acbc878524bfa5e0a220ee37c5d10f6e6a4ca70e08c5624df
Jul 29 20:47:52.368105 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:52.368066    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hnqdd" event={"ID":"f1f85408-f650-4bb2-84c5-6943cefc4c5f","Type":"ContainerStarted","Data":"5bc6736b99887d538186f0950e901a348cd745b15861ec2c723d85c81a3fd36a"}
Jul 29 20:47:52.370311 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:52.370288    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4" event={"ID":"8c4999cf-05ac-4056-82ff-9917e356f0bb","Type":"ContainerStarted","Data":"21679fd793a4c5944d30a64ec8c9ff9f5356ee1566f8c0a9423f287ffd743620"}
Jul 29 20:47:52.370426 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:52.370340    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4" event={"ID":"8c4999cf-05ac-4056-82ff-9917e356f0bb","Type":"ContainerStarted","Data":"2e2f39bf627e224ce54e26179e099f5e2b038368219a5ada774bce33b7913bd0"}
Jul 29 20:47:52.370426 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:52.370367    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4" event={"ID":"8c4999cf-05ac-4056-82ff-9917e356f0bb","Type":"ContainerStarted","Data":"cce1ef2c5e370c5acbc878524bfa5e0a220ee37c5d10f6e6a4ca70e08c5624df"}
Jul 29 20:47:53.375540 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:53.375504    2584 generic.go:356] "Generic (PLEG): container finished" podID="f1f85408-f650-4bb2-84c5-6943cefc4c5f" containerID="aae9cc0ed4a6cdb6e3d73c06c8c571b58d14fa5033ebeb4dbbfd88b6a6bca395" exitCode=0
Jul 29 20:47:53.375966 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:53.375598    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hnqdd" event={"ID":"f1f85408-f650-4bb2-84c5-6943cefc4c5f","Type":"ContainerDied","Data":"aae9cc0ed4a6cdb6e3d73c06c8c571b58d14fa5033ebeb4dbbfd88b6a6bca395"}
Jul 29 20:47:53.377664 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:53.377642    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4" event={"ID":"8c4999cf-05ac-4056-82ff-9917e356f0bb","Type":"ContainerStarted","Data":"e26ebe19cd9bca8a87384737a35f1e65717ecbf121542fbfedea4e1fb3bf8a1e"}
Jul 29 20:47:53.413273 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:53.413219    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-26pn4" podStartSLOduration=2.35390729 podStartE2EDuration="3.413198329s" totalImagesPullingTime="1.059291039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:53.412328303 +0000 UTC m=+89.008620008" watchObservedRunningTime="2026-07-29 20:47:53.413198329 +0000 UTC m=+89.009490014"
Jul 29 20:47:54.382555 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:54.382518    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hnqdd" event={"ID":"f1f85408-f650-4bb2-84c5-6943cefc4c5f","Type":"ContainerStarted","Data":"4359024badf36e1cc5f51f002877773526bd1fa82306de7fa5e08319ad8ba591"}
Jul 29 20:47:54.382947 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:54.382577    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hnqdd" event={"ID":"f1f85408-f650-4bb2-84c5-6943cefc4c5f","Type":"ContainerStarted","Data":"da0ac241b3f9d3c85cd8874b20dbbd2b6aa81263f9254e9715d2d0f7b0308f78"}
Jul 29 20:47:54.399046 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:54.399000    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hnqdd" podStartSLOduration=3.445053638 podStartE2EDuration="4.398984293s" totalImagesPullingTime="882.818901ms" totalInitContainerRuntime="71.111754ms" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:54.397867364 +0000 UTC m=+89.994159049" watchObservedRunningTime="2026-07-29 20:47:54.398984293 +0000 UTC m=+89.995275978"
Jul 29 20:47:56.028700 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.028668    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5f56848b96-sqctc"]
Jul 29 20:47:56.032020 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.032001    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.033875 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.033848    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 29 20:47:56.034003 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.033855    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 29 20:47:56.034003 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.033961    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 29 20:47:56.034121 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.034099    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 29 20:47:56.034183 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.034175    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 29 20:47:56.034232 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.034201    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qj9rv\""
Jul 29 20:47:56.038684 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.038670    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 29 20:47:56.046115 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.046087    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f56848b96-sqctc"]
Jul 29 20:47:56.141347 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.141307    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-metrics-client-ca\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.141528 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.141369    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-secret-telemeter-client\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.141528 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.141400    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-serving-certs-ca-bundle\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.141528 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.141420    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-federate-client-tls\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.141528 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.141444    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcwlx\" (UniqueName: \"kubernetes.io/projected/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-kube-api-access-pcwlx\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.141664 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.141524    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-telemeter-client-tls\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.141664 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.141559    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.141664 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.141585    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.242830 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.242789    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-telemeter-client-tls\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.242830 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.242828    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.243087 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.242856    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.243087 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.242894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-metrics-client-ca\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.243087 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.242945    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-secret-telemeter-client\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.243087 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.242974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-serving-certs-ca-bundle\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.243087 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.242999    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-federate-client-tls\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.243087 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.243022    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcwlx\" (UniqueName: \"kubernetes.io/projected/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-kube-api-access-pcwlx\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.243829 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.243801    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-serving-certs-ca-bundle\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.243946 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.243912    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.244068 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.244010    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-metrics-client-ca\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.245504 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.245476    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.245609 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.245523    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-telemeter-client-tls\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.245609 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.245583    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-secret-telemeter-client\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.245897 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.245874    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-federate-client-tls\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.250278 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.250255    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcwlx\" (UniqueName: \"kubernetes.io/projected/5e7648fb-e8b3-46ff-8bed-1e071c2acd38-kube-api-access-pcwlx\") pod \"telemeter-client-5f56848b96-sqctc\" (UID: \"5e7648fb-e8b3-46ff-8bed-1e071c2acd38\") " pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.341400 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.341312    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc"
Jul 29 20:47:56.468722 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:56.468689    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f56848b96-sqctc"]
Jul 29 20:47:56.472027 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:47:56.472000    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7648fb_e8b3_46ff_8bed_1e071c2acd38.slice/crio-5df45d13f52e740bbac0fa325a2ec2d89be3997ba8b11d33993bec8590494dce WatchSource:0}: Error finding container 5df45d13f52e740bbac0fa325a2ec2d89be3997ba8b11d33993bec8590494dce: Status 404 returned error can't find the container with id 5df45d13f52e740bbac0fa325a2ec2d89be3997ba8b11d33993bec8590494dce
Jul 29 20:47:57.398390 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:57.398358    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc" event={"ID":"5e7648fb-e8b3-46ff-8bed-1e071c2acd38","Type":"ContainerStarted","Data":"5df45d13f52e740bbac0fa325a2ec2d89be3997ba8b11d33993bec8590494dce"}
Jul 29 20:47:59.406644 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:47:59.406599    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc" event={"ID":"5e7648fb-e8b3-46ff-8bed-1e071c2acd38","Type":"ContainerStarted","Data":"d5551b8ee69f98a0f2f74aea126734971966cc596352857ca9b7745f732e63be"}
Jul 29 20:48:00.411612 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:48:00.411575    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc" event={"ID":"5e7648fb-e8b3-46ff-8bed-1e071c2acd38","Type":"ContainerStarted","Data":"cb9525aa5cef60e1ab8fb422d8879003592c56b4616922227ab49901de6d0556"}
Jul 29 20:48:00.412086 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:48:00.411641    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc" event={"ID":"5e7648fb-e8b3-46ff-8bed-1e071c2acd38","Type":"ContainerStarted","Data":"c410e38d69feccf9e8b1c6c51b0f1886089aaa98cf1d71a569779d14948efaff"}
Jul 29 20:48:00.430291 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:48:00.430244    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5f56848b96-sqctc" podStartSLOduration=1.578877259 podStartE2EDuration="4.430224625s" totalImagesPullingTime="2.851347366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:48:00.428618777 +0000 UTC m=+96.024910461" watchObservedRunningTime="2026-07-29 20:48:00.430224625 +0000 UTC m=+96.026516310"
Jul 29 20:48:12.334428 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:48:12.334398    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v5zlw"
Jul 29 20:48:24.857997 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:48:24.857970    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qnhjk_5c357a9f-b01a-41c9-b36d-cdd506c9b60d/serve-healthcheck-canary/0.log"
Jul 29 20:50:48.423743 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.423657    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qjg6h"]
Jul 29 20:50:48.426692 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.426677    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qjg6h"
Jul 29 20:50:48.428767 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.428746    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 29 20:50:48.428884 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.428788    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 29 20:50:48.429345 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.429329    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6hdgz\""
Jul 29 20:50:48.435999 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.435975    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qjg6h"]
Jul 29 20:50:48.442024 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.442002    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh4mm\" (UniqueName: \"kubernetes.io/projected/c29cd455-44bd-47c0-9c28-31aee1b4da82-kube-api-access-fh4mm\") pod \"cert-manager-webhook-6486ffd997-qjg6h\" (UID: \"c29cd455-44bd-47c0-9c28-31aee1b4da82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qjg6h"
Jul 29 20:50:48.442121 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.442032    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c29cd455-44bd-47c0-9c28-31aee1b4da82-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qjg6h\" (UID: \"c29cd455-44bd-47c0-9c28-31aee1b4da82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qjg6h"
Jul 29 20:50:48.543452 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.543413    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh4mm\" (UniqueName: \"kubernetes.io/projected/c29cd455-44bd-47c0-9c28-31aee1b4da82-kube-api-access-fh4mm\") pod \"cert-manager-webhook-6486ffd997-qjg6h\" (UID: \"c29cd455-44bd-47c0-9c28-31aee1b4da82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qjg6h"
Jul 29 20:50:48.543452 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.543457    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c29cd455-44bd-47c0-9c28-31aee1b4da82-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qjg6h\" (UID: \"c29cd455-44bd-47c0-9c28-31aee1b4da82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qjg6h"
Jul 29 20:50:48.551399 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.551372    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c29cd455-44bd-47c0-9c28-31aee1b4da82-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qjg6h\" (UID: \"c29cd455-44bd-47c0-9c28-31aee1b4da82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qjg6h"
Jul 29 20:50:48.551510 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.551432    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh4mm\" (UniqueName: \"kubernetes.io/projected/c29cd455-44bd-47c0-9c28-31aee1b4da82-kube-api-access-fh4mm\") pod \"cert-manager-webhook-6486ffd997-qjg6h\" (UID: \"c29cd455-44bd-47c0-9c28-31aee1b4da82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qjg6h"
Jul 29 20:50:48.735440 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.735354    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qjg6h"
Jul 29 20:50:48.857033 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.857009    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qjg6h"]
Jul 29 20:50:48.859542 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:50:48.859509    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc29cd455_44bd_47c0_9c28_31aee1b4da82.slice/crio-92aecabdaec853adab837832ee1de3e62c829eea8f4da331be7e3e71f9f5c02d WatchSource:0}: Error finding container 92aecabdaec853adab837832ee1de3e62c829eea8f4da331be7e3e71f9f5c02d: Status 404 returned error can't find the container with id 92aecabdaec853adab837832ee1de3e62c829eea8f4da331be7e3e71f9f5c02d
Jul 29 20:50:48.962514 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:48.962481    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qjg6h" event={"ID":"c29cd455-44bd-47c0-9c28-31aee1b4da82","Type":"ContainerStarted","Data":"92aecabdaec853adab837832ee1de3e62c829eea8f4da331be7e3e71f9f5c02d"}
Jul 29 20:50:52.979285 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:52.979241    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qjg6h" event={"ID":"c29cd455-44bd-47c0-9c28-31aee1b4da82","Type":"ContainerStarted","Data":"023c0f8e4e9c5de35851c7e3eb64d02c8d38af36d53ee034a9a74b5234e1adf7"}
Jul 29 20:50:52.979695 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:52.979358    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qjg6h"
Jul 29 20:50:52.993931 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:52.993887    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-qjg6h" podStartSLOduration=1.7704675829999998 podStartE2EDuration="4.993875134s" totalImagesPullingTime="3.223407551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:50:52.993522501 +0000 UTC m=+268.589814188" watchObservedRunningTime="2026-07-29 20:50:52.993875134 +0000 UTC m=+268.590166818"
Jul 29 20:50:58.984671 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:50:58.984641    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qjg6h"
Jul 29 20:51:14.708919 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.708887    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"]
Jul 29 20:51:14.712128 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.712100    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.714136 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.714114    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 20:51:14.714276 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.714167    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x7p5x\""
Jul 29 20:51:14.714400 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.714387    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 20:51:14.715054 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.715033    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 20:51:14.715843 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.715828    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 20:51:14.732904 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.732883    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"]
Jul 29 20:51:14.826360 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.826340    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szn76\" (UniqueName: \"kubernetes.io/projected/f820a960-882d-488f-8de5-3aaa08ace55f-kube-api-access-szn76\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.826476 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.826373    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f820a960-882d-488f-8de5-3aaa08ace55f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.826476 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.826450    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f820a960-882d-488f-8de5-3aaa08ace55f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.826549 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.826484    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f820a960-882d-488f-8de5-3aaa08ace55f-tmp\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.826549 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.826518    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f820a960-882d-488f-8de5-3aaa08ace55f-manifests\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.927501 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.927476    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f820a960-882d-488f-8de5-3aaa08ace55f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.927629 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.927506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f820a960-882d-488f-8de5-3aaa08ace55f-tmp\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.927629 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.927540    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f820a960-882d-488f-8de5-3aaa08ace55f-manifests\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.927629 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.927597    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szn76\" (UniqueName: \"kubernetes.io/projected/f820a960-882d-488f-8de5-3aaa08ace55f-kube-api-access-szn76\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.927748 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.927649    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f820a960-882d-488f-8de5-3aaa08ace55f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.928109 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.928089    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f820a960-882d-488f-8de5-3aaa08ace55f-manifests\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.929808 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.929790    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f820a960-882d-488f-8de5-3aaa08ace55f-tmp\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.929935 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.929919    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f820a960-882d-488f-8de5-3aaa08ace55f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.930036 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.930013    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f820a960-882d-488f-8de5-3aaa08ace55f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:14.935283 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:14.935261    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szn76\" (UniqueName: \"kubernetes.io/projected/f820a960-882d-488f-8de5-3aaa08ace55f-kube-api-access-szn76\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-jgz46\" (UID: \"f820a960-882d-488f-8de5-3aaa08ace55f\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:15.021957 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:15.021901    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:15.147614 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:15.147587    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"]
Jul 29 20:51:15.150563 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:51:15.150536    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf820a960_882d_488f_8de5_3aaa08ace55f.slice/crio-d2ed62e53d39e85e4972fb64d90377936d29df7e83bca54f5c55cb3a2ca8cc52 WatchSource:0}: Error finding container d2ed62e53d39e85e4972fb64d90377936d29df7e83bca54f5c55cb3a2ca8cc52: Status 404 returned error can't find the container with id d2ed62e53d39e85e4972fb64d90377936d29df7e83bca54f5c55cb3a2ca8cc52
Jul 29 20:51:16.057137 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:16.057080    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46" event={"ID":"f820a960-882d-488f-8de5-3aaa08ace55f","Type":"ContainerStarted","Data":"d2ed62e53d39e85e4972fb64d90377936d29df7e83bca54f5c55cb3a2ca8cc52"}
Jul 29 20:51:19.073428 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:19.073395    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46" event={"ID":"f820a960-882d-488f-8de5-3aaa08ace55f","Type":"ContainerStarted","Data":"0ce9f85c72b8d6da2e49268f300a0e78e8341fa5122524fb6dcd507314fe2bc4"}
Jul 29 20:51:20.078605 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:20.078564    2584 generic.go:356] "Generic (PLEG): container finished" podID="f820a960-882d-488f-8de5-3aaa08ace55f" containerID="0ce9f85c72b8d6da2e49268f300a0e78e8341fa5122524fb6dcd507314fe2bc4" exitCode=0
Jul 29 20:51:20.079070 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:20.078641    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46" event={"ID":"f820a960-882d-488f-8de5-3aaa08ace55f","Type":"ContainerDied","Data":"0ce9f85c72b8d6da2e49268f300a0e78e8341fa5122524fb6dcd507314fe2bc4"}
Jul 29 20:51:21.083545 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:21.083517    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46" event={"ID":"f820a960-882d-488f-8de5-3aaa08ace55f","Type":"ContainerStarted","Data":"aa0140dd310dd9bbcc836a50665b4cb88309a6dfc11efa804c33c9b4cb7dd040"}
Jul 29 20:51:21.083924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:21.083645    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:21.108668 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:21.108623    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46" podStartSLOduration=2.725283485 podStartE2EDuration="7.108611054s" totalImagesPullingTime="3.32931541s" totalInitContainerRuntime="1.054012159s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:21.10732916 +0000 UTC m=+296.703620844" watchObservedRunningTime="2026-07-29 20:51:21.108611054 +0000 UTC m=+296.704902763"
Jul 29 20:51:24.739578 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:24.739545    2584 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 20:51:32.088986 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.088950    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-jgz46"
Jul 29 20:51:32.218075 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.218034    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"]
Jul 29 20:51:32.220071 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.220055    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.222099 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.222074    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 20:51:32.222099 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.222085    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 20:51:32.222274 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.222226    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 20:51:32.222365 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.222347    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mhtvw\""
Jul 29 20:51:32.222425 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.222378    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:51:32.222487 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.222430    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 20:51:32.229624 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.229605    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"]
Jul 29 20:51:32.360861 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.360831    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e26c4b45-4911-482f-9815-e4c1140b68e4-metrics-cert\") pod \"lws-controller-manager-6896cc89b8-kfd7t\" (UID: \"e26c4b45-4911-482f-9815-e4c1140b68e4\") " pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.361010 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.360889    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q85qs\" (UniqueName: \"kubernetes.io/projected/e26c4b45-4911-482f-9815-e4c1140b68e4-kube-api-access-q85qs\") pod \"lws-controller-manager-6896cc89b8-kfd7t\" (UID: \"e26c4b45-4911-482f-9815-e4c1140b68e4\") " pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.361010 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.360945    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e26c4b45-4911-482f-9815-e4c1140b68e4-manager-config\") pod \"lws-controller-manager-6896cc89b8-kfd7t\" (UID: \"e26c4b45-4911-482f-9815-e4c1140b68e4\") " pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.361010 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.360979    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e26c4b45-4911-482f-9815-e4c1140b68e4-cert\") pod \"lws-controller-manager-6896cc89b8-kfd7t\" (UID: \"e26c4b45-4911-482f-9815-e4c1140b68e4\") " pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.462157 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.462093    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q85qs\" (UniqueName: \"kubernetes.io/projected/e26c4b45-4911-482f-9815-e4c1140b68e4-kube-api-access-q85qs\") pod \"lws-controller-manager-6896cc89b8-kfd7t\" (UID: \"e26c4b45-4911-482f-9815-e4c1140b68e4\") " pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.462157 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.462165    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e26c4b45-4911-482f-9815-e4c1140b68e4-manager-config\") pod \"lws-controller-manager-6896cc89b8-kfd7t\" (UID: \"e26c4b45-4911-482f-9815-e4c1140b68e4\") " pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.462410 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.462192    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e26c4b45-4911-482f-9815-e4c1140b68e4-cert\") pod \"lws-controller-manager-6896cc89b8-kfd7t\" (UID: \"e26c4b45-4911-482f-9815-e4c1140b68e4\") " pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.462410 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.462249    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e26c4b45-4911-482f-9815-e4c1140b68e4-metrics-cert\") pod \"lws-controller-manager-6896cc89b8-kfd7t\" (UID: \"e26c4b45-4911-482f-9815-e4c1140b68e4\") " pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.462899 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.462878    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e26c4b45-4911-482f-9815-e4c1140b68e4-manager-config\") pod \"lws-controller-manager-6896cc89b8-kfd7t\" (UID: \"e26c4b45-4911-482f-9815-e4c1140b68e4\") " pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.464636 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.464615    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e26c4b45-4911-482f-9815-e4c1140b68e4-metrics-cert\") pod \"lws-controller-manager-6896cc89b8-kfd7t\" (UID: \"e26c4b45-4911-482f-9815-e4c1140b68e4\") " pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.464804 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.464784    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e26c4b45-4911-482f-9815-e4c1140b68e4-cert\") pod \"lws-controller-manager-6896cc89b8-kfd7t\" (UID: \"e26c4b45-4911-482f-9815-e4c1140b68e4\") " pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.470171 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.470132    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q85qs\" (UniqueName: \"kubernetes.io/projected/e26c4b45-4911-482f-9815-e4c1140b68e4-kube-api-access-q85qs\") pod \"lws-controller-manager-6896cc89b8-kfd7t\" (UID: \"e26c4b45-4911-482f-9815-e4c1140b68e4\") " pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.529483 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.529456    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:32.655984 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.655924    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"]
Jul 29 20:51:32.658789 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:51:32.658765    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26c4b45_4911_482f_9815_e4c1140b68e4.slice/crio-a71253aec936207cad4873ee5108ade077a74307ef7ad4f3b5eb4ec1cb2ef375 WatchSource:0}: Error finding container a71253aec936207cad4873ee5108ade077a74307ef7ad4f3b5eb4ec1cb2ef375: Status 404 returned error can't find the container with id a71253aec936207cad4873ee5108ade077a74307ef7ad4f3b5eb4ec1cb2ef375
Jul 29 20:51:32.660672 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:32.660656    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:51:33.124157 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:33.124113    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t" event={"ID":"e26c4b45-4911-482f-9815-e4c1140b68e4","Type":"ContainerStarted","Data":"a71253aec936207cad4873ee5108ade077a74307ef7ad4f3b5eb4ec1cb2ef375"}
Jul 29 20:51:36.136304 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:36.136271    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t" event={"ID":"e26c4b45-4911-482f-9815-e4c1140b68e4","Type":"ContainerStarted","Data":"bd8d2ffda506fee2543cb8af737771f34f3168ac4af998aa4a3cbf0f86e7a37c"}
Jul 29 20:51:36.136657 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:36.136400    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:36.153195 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:36.153132    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t" podStartSLOduration=1.689872904 podStartE2EDuration="4.153117997s" totalImagesPullingTime="2.463245093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:36.151994352 +0000 UTC m=+311.748286053" watchObservedRunningTime="2026-07-29 20:51:36.153117997 +0000 UTC m=+311.749409682"
Jul 29 20:51:41.947408 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:41.947372    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"]
Jul 29 20:51:41.951903 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:41.951876    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:41.954637 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:41.954617    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 20:51:41.954970 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:41.954954    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-6r2r2\""
Jul 29 20:51:41.955311 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:41.955295    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 20:51:41.968320 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:41.968296    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 20:51:41.973066 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:41.973045    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"]
Jul 29 20:51:42.036724 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.036697    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.036724 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.036727    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fafa176-2306-4a25-8301-c47c104c43ab-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.036930 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.036751    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.036930 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.036780    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fafa176-2306-4a25-8301-c47c104c43ab-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.036930 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.036852    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.036930 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.036892    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccg49\" (UniqueName: \"kubernetes.io/projected/9fafa176-2306-4a25-8301-c47c104c43ab-kube-api-access-ccg49\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.036930 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.036913    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.037105 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.036933    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fafa176-2306-4a25-8301-c47c104c43ab-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.037105 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.036962    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.066055 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.066024    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"]
Jul 29 20:51:42.068864 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.068846    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.081450 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.081429    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"]
Jul 29 20:51:42.137425 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.137401    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fafa176-2306-4a25-8301-c47c104c43ab-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.137603 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.137440    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.137603 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.137475    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccg49\" (UniqueName: \"kubernetes.io/projected/9fafa176-2306-4a25-8301-c47c104c43ab-kube-api-access-ccg49\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.137603 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.137503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.137603 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.137529    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fafa176-2306-4a25-8301-c47c104c43ab-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.137603 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.137572    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.137872 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.137634    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.137948 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.137927    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.138011 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.137993    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fafa176-2306-4a25-8301-c47c104c43ab-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.138063 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.138041    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.138175 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.138132    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.138356 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.138258    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fafa176-2306-4a25-8301-c47c104c43ab-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.138496 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.138475    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.138570 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.138539    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.140133 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.140115    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.140429 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.140413    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fafa176-2306-4a25-8301-c47c104c43ab-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.144834 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.144811    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fafa176-2306-4a25-8301-c47c104c43ab-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.146522 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.146500    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccg49\" (UniqueName: \"kubernetes.io/projected/9fafa176-2306-4a25-8301-c47c104c43ab-kube-api-access-ccg49\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcw64ft\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.239366 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.239290    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e266ed-b7ab-4356-83c4-b942b5350187-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.239366 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.239333    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.239366 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.239367    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.239556 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.239389    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.239556 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.239412    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e266ed-b7ab-4356-83c4-b942b5350187-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.239556 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.239489    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x94lp\" (UniqueName: \"kubernetes.io/projected/e8e266ed-b7ab-4356-83c4-b942b5350187-kube-api-access-x94lp\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.239556 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.239521    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.239556 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.239544    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.239737 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.239626    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e266ed-b7ab-4356-83c4-b942b5350187-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.261394 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.261359    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:42.340690 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.340655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.340828 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.340713    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.340828 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.340748    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e266ed-b7ab-4356-83c4-b942b5350187-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.340828 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.340799    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94lp\" (UniqueName: \"kubernetes.io/projected/e8e266ed-b7ab-4356-83c4-b942b5350187-kube-api-access-x94lp\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.340970 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.340835    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.340970 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.340872    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.340970 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.340929    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e266ed-b7ab-4356-83c4-b942b5350187-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.341060 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.340973    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e266ed-b7ab-4356-83c4-b942b5350187-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.341060 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.341011    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.341133 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.341108    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.341401 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.341380    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.341972 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.341649    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.342276 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.342233    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.342386 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.342284    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e266ed-b7ab-4356-83c4-b942b5350187-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.343925 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.343904    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e266ed-b7ab-4356-83c4-b942b5350187-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.344251 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.344228    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e266ed-b7ab-4356-83c4-b942b5350187-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.349201 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.349138    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94lp\" (UniqueName: \"kubernetes.io/projected/e8e266ed-b7ab-4356-83c4-b942b5350187-kube-api-access-x94lp\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.349310 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.349288    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e266ed-b7ab-4356-83c4-b942b5350187-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc\" (UID: \"e8e266ed-b7ab-4356-83c4-b942b5350187\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.378390 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.378361    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:42.390589 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.390564    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"]
Jul 29 20:51:42.391436 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:51:42.391414    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fafa176_2306_4a25_8301_c47c104c43ab.slice/crio-f25c931a66f4efe49e768d30c514312fd2e94304c72d72709bfe1c729e7fa9e3 WatchSource:0}: Error finding container f25c931a66f4efe49e768d30c514312fd2e94304c72d72709bfe1c729e7fa9e3: Status 404 returned error can't find the container with id f25c931a66f4efe49e768d30c514312fd2e94304c72d72709bfe1c729e7fa9e3
Jul 29 20:51:42.538607 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:42.538577    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"]
Jul 29 20:51:42.540876 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:51:42.540851    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e266ed_b7ab_4356_83c4_b942b5350187.slice/crio-ddef3270fe1aa1a1e584c4934147090006e2aaa03f85a6e8c7b842382f625848 WatchSource:0}: Error finding container ddef3270fe1aa1a1e584c4934147090006e2aaa03f85a6e8c7b842382f625848: Status 404 returned error can't find the container with id ddef3270fe1aa1a1e584c4934147090006e2aaa03f85a6e8c7b842382f625848
Jul 29 20:51:43.160943 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:43.160910    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft" event={"ID":"9fafa176-2306-4a25-8301-c47c104c43ab","Type":"ContainerStarted","Data":"f25c931a66f4efe49e768d30c514312fd2e94304c72d72709bfe1c729e7fa9e3"}
Jul 29 20:51:43.162020 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:43.162001    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc" event={"ID":"e8e266ed-b7ab-4356-83c4-b942b5350187","Type":"ContainerStarted","Data":"ddef3270fe1aa1a1e584c4934147090006e2aaa03f85a6e8c7b842382f625848"}
Jul 29 20:51:43.272204 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:51:43.272158    2584 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/32/32545ef0237a95c1159d8f7f8d880bac7fc5d595895a70e4528ec30f8ff67926?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260729%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260729T205143Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=6b2d24fb73d17046eb47803da6a5dd12b3ab32feb3cb9d21e6f4180bb59f4205&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1785359203~hmac=fc1e94cd62b9508bc6720df81e21fa12cf6335fdef3d2fc64081941dddc39632\": EOF; artifact err: provided artifact is a container image" image="registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec"
Jul 29 20:51:43.272375 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:51:43.272276    2584 kuberuntime_manager.go:1459] "Unhandled Error" err="container istio-proxy start failed in pod data-science-gateway-data-science-gateway-class-6745bb66bcw64ft_openshift-ingress(9fafa176-2306-4a25-8301-c47c104c43ab): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/32/32545ef0237a95c1159d8f7f8d880bac7fc5d595895a70e4528ec30f8ff67926?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260729%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260729T205143Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=6b2d24fb73d17046eb47803da6a5dd12b3ab32feb3cb9d21e6f4180bb59f4205&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1785359203~hmac=fc1e94cd62b9508bc6720df81e21fa12cf6335fdef3d2fc64081941dddc39632\": EOF; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 29 20:51:43.272375 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:51:43.272318    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/32/32545ef0237a95c1159d8f7f8d880bac7fc5d595895a70e4528ec30f8ff67926?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260729%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260729T205143Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=6b2d24fb73d17046eb47803da6a5dd12b3ab32feb3cb9d21e6f4180bb59f4205&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1785359203~hmac=fc1e94cd62b9508bc6720df81e21fa12cf6335fdef3d2fc64081941dddc39632\\\": EOF; artifact err: provided artifact is a container image\"" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft" podUID="9fafa176-2306-4a25-8301-c47c104c43ab"
Jul 29 20:51:44.168236 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:51:44.168176    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/32/32545ef0237a95c1159d8f7f8d880bac7fc5d595895a70e4528ec30f8ff67926?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260729%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260729T205143Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=6b2d24fb73d17046eb47803da6a5dd12b3ab32feb3cb9d21e6f4180bb59f4205&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1785359203~hmac=fc1e94cd62b9508bc6720df81e21fa12cf6335fdef3d2fc64081941dddc39632\\\": EOF; artifact err: provided artifact is a container image\"" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft" podUID="9fafa176-2306-4a25-8301-c47c104c43ab"
Jul 29 20:51:45.905923 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:45.905890    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"]
Jul 29 20:51:45.916243 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:45.916211    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z"]
Jul 29 20:51:45.916376 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:45.916326    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"
Jul 29 20:51:45.918958 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:45.918932    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 29 20:51:45.919087 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:45.918941    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-h65hf\""
Jul 29 20:51:45.921042 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:45.921023    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z"
Jul 29 20:51:45.923536 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:45.923197    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-92mwb\""
Jul 29 20:51:45.923536 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:45.923403    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 29 20:51:45.924019 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:45.923999    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"]
Jul 29 20:51:45.941923 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:45.941890    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z"]
Jul 29 20:51:45.999233 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:45.999205    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 20:51:45.999315 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:45.999267    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 20:51:45.999315 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:45.999304    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 20:51:46.072752 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.072724    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/779aa379-51b0-430b-97f2-7f53d397fcec-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lgk7z\" (UID: \"779aa379-51b0-430b-97f2-7f53d397fcec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z"
Jul 29 20:51:46.072835 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.072776    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/939915a3-8bc5-4be2-a221-f7ca7ff44d94-cert\") pod \"kserve-controller-manager-d6f4b65b8-8x7hm\" (UID: \"939915a3-8bc5-4be2-a221-f7ca7ff44d94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"
Jul 29 20:51:46.072835 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.072823    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wh2\" (UniqueName: \"kubernetes.io/projected/779aa379-51b0-430b-97f2-7f53d397fcec-kube-api-access-g2wh2\") pod \"llmisvc-controller-manager-777c6bd7d5-lgk7z\" (UID: \"779aa379-51b0-430b-97f2-7f53d397fcec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z"
Jul 29 20:51:46.072949 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.072903    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mqqx\" (UniqueName: \"kubernetes.io/projected/939915a3-8bc5-4be2-a221-f7ca7ff44d94-kube-api-access-6mqqx\") pod \"kserve-controller-manager-d6f4b65b8-8x7hm\" (UID: \"939915a3-8bc5-4be2-a221-f7ca7ff44d94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"
Jul 29 20:51:46.174447 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.174352    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/779aa379-51b0-430b-97f2-7f53d397fcec-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lgk7z\" (UID: \"779aa379-51b0-430b-97f2-7f53d397fcec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z"
Jul 29 20:51:46.174447 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.174430    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/939915a3-8bc5-4be2-a221-f7ca7ff44d94-cert\") pod \"kserve-controller-manager-d6f4b65b8-8x7hm\" (UID: \"939915a3-8bc5-4be2-a221-f7ca7ff44d94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"
Jul 29 20:51:46.174662 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.174461    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wh2\" (UniqueName: \"kubernetes.io/projected/779aa379-51b0-430b-97f2-7f53d397fcec-kube-api-access-g2wh2\") pod \"llmisvc-controller-manager-777c6bd7d5-lgk7z\" (UID: \"779aa379-51b0-430b-97f2-7f53d397fcec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z"
Jul 29 20:51:46.174662 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.174528    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mqqx\" (UniqueName: \"kubernetes.io/projected/939915a3-8bc5-4be2-a221-f7ca7ff44d94-kube-api-access-6mqqx\") pod \"kserve-controller-manager-d6f4b65b8-8x7hm\" (UID: \"939915a3-8bc5-4be2-a221-f7ca7ff44d94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"
Jul 29 20:51:46.174662 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:51:46.174600    2584 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 29 20:51:46.174815 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:51:46.174677    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/939915a3-8bc5-4be2-a221-f7ca7ff44d94-cert podName:939915a3-8bc5-4be2-a221-f7ca7ff44d94 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:46.674655283 +0000 UTC m=+322.270946966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/939915a3-8bc5-4be2-a221-f7ca7ff44d94-cert") pod "kserve-controller-manager-d6f4b65b8-8x7hm" (UID: "939915a3-8bc5-4be2-a221-f7ca7ff44d94") : secret "kserve-webhook-server-cert" not found
Jul 29 20:51:46.177104 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.177077    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/779aa379-51b0-430b-97f2-7f53d397fcec-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lgk7z\" (UID: \"779aa379-51b0-430b-97f2-7f53d397fcec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z"
Jul 29 20:51:46.179946 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.179920    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc" event={"ID":"e8e266ed-b7ab-4356-83c4-b942b5350187","Type":"ContainerStarted","Data":"9b40dd53f1927591b378ca73cde31dab1d8ff594bc5ab5e647060e14bce409ce"}
Jul 29 20:51:46.184292 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.184271    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wh2\" (UniqueName: \"kubernetes.io/projected/779aa379-51b0-430b-97f2-7f53d397fcec-kube-api-access-g2wh2\") pod \"llmisvc-controller-manager-777c6bd7d5-lgk7z\" (UID: \"779aa379-51b0-430b-97f2-7f53d397fcec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z"
Jul 29 20:51:46.184609 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.184592    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mqqx\" (UniqueName: \"kubernetes.io/projected/939915a3-8bc5-4be2-a221-f7ca7ff44d94-kube-api-access-6mqqx\") pod \"kserve-controller-manager-d6f4b65b8-8x7hm\" (UID: \"939915a3-8bc5-4be2-a221-f7ca7ff44d94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"
Jul 29 20:51:46.199888 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.199838    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc" podStartSLOduration=0.743641494 podStartE2EDuration="4.199820106s" totalImagesPullingTime="3.456178612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:46.198981101 +0000 UTC m=+321.795272837" watchObservedRunningTime="2026-07-29 20:51:46.199820106 +0000 UTC m=+321.796111791"
Jul 29 20:51:46.236274 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.236253    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z"
Jul 29 20:51:46.367381 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.367354    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z"]
Jul 29 20:51:46.369809 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:51:46.369784    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod779aa379_51b0_430b_97f2_7f53d397fcec.slice/crio-2a4b94fd9a3154c3290b44f836ee3a6e8da41e5e270e843de32f36e6059f622b WatchSource:0}: Error finding container 2a4b94fd9a3154c3290b44f836ee3a6e8da41e5e270e843de32f36e6059f622b: Status 404 returned error can't find the container with id 2a4b94fd9a3154c3290b44f836ee3a6e8da41e5e270e843de32f36e6059f622b
Jul 29 20:51:46.378902 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.378880    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:46.678512 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.678477    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/939915a3-8bc5-4be2-a221-f7ca7ff44d94-cert\") pod \"kserve-controller-manager-d6f4b65b8-8x7hm\" (UID: \"939915a3-8bc5-4be2-a221-f7ca7ff44d94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"
Jul 29 20:51:46.681224 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.681205    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/939915a3-8bc5-4be2-a221-f7ca7ff44d94-cert\") pod \"kserve-controller-manager-d6f4b65b8-8x7hm\" (UID: \"939915a3-8bc5-4be2-a221-f7ca7ff44d94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"
Jul 29 20:51:46.830322 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.830292    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"
Jul 29 20:51:46.954716 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:46.954690    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"]
Jul 29 20:51:46.956929 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:51:46.956903    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939915a3_8bc5_4be2_a221_f7ca7ff44d94.slice/crio-b3e082623d94623c78bde44a1ba00facdc6300a605e5ca6f7f7054c8578253f1 WatchSource:0}: Error finding container b3e082623d94623c78bde44a1ba00facdc6300a605e5ca6f7f7054c8578253f1: Status 404 returned error can't find the container with id b3e082623d94623c78bde44a1ba00facdc6300a605e5ca6f7f7054c8578253f1
Jul 29 20:51:47.141871 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:47.141842    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6896cc89b8-kfd7t"
Jul 29 20:51:47.186021 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:47.185977    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm" event={"ID":"939915a3-8bc5-4be2-a221-f7ca7ff44d94","Type":"ContainerStarted","Data":"b3e082623d94623c78bde44a1ba00facdc6300a605e5ca6f7f7054c8578253f1"}
Jul 29 20:51:47.187406 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:47.187378    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z" event={"ID":"779aa379-51b0-430b-97f2-7f53d397fcec","Type":"ContainerStarted","Data":"2a4b94fd9a3154c3290b44f836ee3a6e8da41e5e270e843de32f36e6059f622b"}
Jul 29 20:51:47.384689 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:47.384663    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:48.190767 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:48.190735    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:48.191776 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:48.191754    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc"
Jul 29 20:51:48.301430 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:48.301400    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"]
Jul 29 20:51:48.909346 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:48.909318    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:49.101136 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101099    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fafa176-2306-4a25-8301-c47c104c43ab-istiod-ca-cert\") pod \"9fafa176-2306-4a25-8301-c47c104c43ab\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") "
Jul 29 20:51:49.101375 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101159    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-workload-certs\") pod \"9fafa176-2306-4a25-8301-c47c104c43ab\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") "
Jul 29 20:51:49.101375 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101205    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-credential-socket\") pod \"9fafa176-2306-4a25-8301-c47c104c43ab\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") "
Jul 29 20:51:49.101375 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101235    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-istio-data\") pod \"9fafa176-2306-4a25-8301-c47c104c43ab\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") "
Jul 29 20:51:49.101375 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101270    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-istio-envoy\") pod \"9fafa176-2306-4a25-8301-c47c104c43ab\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") "
Jul 29 20:51:49.101375 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101299    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fafa176-2306-4a25-8301-c47c104c43ab-istio-podinfo\") pod \"9fafa176-2306-4a25-8301-c47c104c43ab\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") "
Jul 29 20:51:49.101375 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101358    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccg49\" (UniqueName: \"kubernetes.io/projected/9fafa176-2306-4a25-8301-c47c104c43ab-kube-api-access-ccg49\") pod \"9fafa176-2306-4a25-8301-c47c104c43ab\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") "
Jul 29 20:51:49.101673 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101401    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-workload-socket\") pod \"9fafa176-2306-4a25-8301-c47c104c43ab\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") "
Jul 29 20:51:49.101673 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101450    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fafa176-2306-4a25-8301-c47c104c43ab-istio-token\") pod \"9fafa176-2306-4a25-8301-c47c104c43ab\" (UID: \"9fafa176-2306-4a25-8301-c47c104c43ab\") "
Jul 29 20:51:49.101673 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101481    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9fafa176-2306-4a25-8301-c47c104c43ab" (UID: "9fafa176-2306-4a25-8301-c47c104c43ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:51:49.101673 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101586    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fafa176-2306-4a25-8301-c47c104c43ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9fafa176-2306-4a25-8301-c47c104c43ab" (UID: "9fafa176-2306-4a25-8301-c47c104c43ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:51:49.101864 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101720    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fafa176-2306-4a25-8301-c47c104c43ab-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:51:49.101864 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101742    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:51:49.101864 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101845    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9fafa176-2306-4a25-8301-c47c104c43ab" (UID: "9fafa176-2306-4a25-8301-c47c104c43ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:51:49.102003 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101925    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9fafa176-2306-4a25-8301-c47c104c43ab" (UID: "9fafa176-2306-4a25-8301-c47c104c43ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:51:49.102003 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.101985    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "9fafa176-2306-4a25-8301-c47c104c43ab" (UID: "9fafa176-2306-4a25-8301-c47c104c43ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:51:49.104013 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.103959    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fafa176-2306-4a25-8301-c47c104c43ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9fafa176-2306-4a25-8301-c47c104c43ab" (UID: "9fafa176-2306-4a25-8301-c47c104c43ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 20:51:49.104132 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.104079    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fafa176-2306-4a25-8301-c47c104c43ab-kube-api-access-ccg49" (OuterVolumeSpecName: "kube-api-access-ccg49") pod "9fafa176-2306-4a25-8301-c47c104c43ab" (UID: "9fafa176-2306-4a25-8301-c47c104c43ab"). InnerVolumeSpecName "kube-api-access-ccg49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:51:49.104282 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.104250    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9fafa176-2306-4a25-8301-c47c104c43ab" (UID: "9fafa176-2306-4a25-8301-c47c104c43ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:51:49.104385 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.104345    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fafa176-2306-4a25-8301-c47c104c43ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "9fafa176-2306-4a25-8301-c47c104c43ab" (UID: "9fafa176-2306-4a25-8301-c47c104c43ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:51:49.196174 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.196077    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft" event={"ID":"9fafa176-2306-4a25-8301-c47c104c43ab","Type":"ContainerDied","Data":"f25c931a66f4efe49e768d30c514312fd2e94304c72d72709bfe1c729e7fa9e3"}
Jul 29 20:51:49.196174 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.196096    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"
Jul 29 20:51:49.202819 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.202795    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:51:49.202924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.202820    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:51:49.202924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.202834    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:51:49.202924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.202847    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fafa176-2306-4a25-8301-c47c104c43ab-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:51:49.202924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.202859    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccg49\" (UniqueName: \"kubernetes.io/projected/9fafa176-2306-4a25-8301-c47c104c43ab-kube-api-access-ccg49\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:51:49.202924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.202874    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafa176-2306-4a25-8301-c47c104c43ab-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:51:49.202924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.202888    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fafa176-2306-4a25-8301-c47c104c43ab-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:51:49.234566 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.234530    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"]
Jul 29 20:51:49.244769 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:49.244730    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcw64ft"]
Jul 29 20:51:50.201386 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:50.201352    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm" event={"ID":"939915a3-8bc5-4be2-a221-f7ca7ff44d94","Type":"ContainerStarted","Data":"a3dbb39a876c97ea6c265f5ee7547dad8facaaaa61a178668648a9f701f23c0a"}
Jul 29 20:51:50.201821 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:50.201436    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"
Jul 29 20:51:50.202829 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:50.202803    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z" event={"ID":"779aa379-51b0-430b-97f2-7f53d397fcec","Type":"ContainerStarted","Data":"50301e13039e0a7fdd5963009d21838207ca97c04a7e50be06b09c52524f2748"}
Jul 29 20:51:50.202934 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:50.202894    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z"
Jul 29 20:51:50.215561 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:50.215519    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm" podStartSLOduration=2.6929723660000002 podStartE2EDuration="5.215505877s" totalImagesPullingTime="2.522533511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:50.214608373 +0000 UTC m=+325.810900057" watchObservedRunningTime="2026-07-29 20:51:50.215505877 +0000 UTC m=+325.811797561"
Jul 29 20:51:50.229767 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:50.229728    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z" podStartSLOduration=2.122077342 podStartE2EDuration="5.229716393s" totalImagesPullingTime="3.107639051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:50.228320806 +0000 UTC m=+325.824612490" watchObservedRunningTime="2026-07-29 20:51:50.229716393 +0000 UTC m=+325.826008078"
Jul 29 20:51:50.864330 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:51:50.864303    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fafa176-2306-4a25-8301-c47c104c43ab" path="/var/lib/kubelet/pods/9fafa176-2306-4a25-8301-c47c104c43ab/volumes"
Jul 29 20:52:21.208902 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:52:21.208830    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lgk7z"
Jul 29 20:52:21.211941 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:52:21.211920    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8x7hm"
Jul 29 20:53:16.253836 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:16.253797    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5wrts"]
Jul 29 20:53:16.256313 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:16.256294    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5wrts"
Jul 29 20:53:16.259561 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:16.259538    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 20:53:16.259667 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:16.259544    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 20:53:16.259832 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:16.259814    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2hn4f\""
Jul 29 20:53:16.265435 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:16.265412    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk2n2\" (UniqueName: \"kubernetes.io/projected/b406c2b2-6add-4e70-bdaa-c358471c2144-kube-api-access-hk2n2\") pod \"authorino-operator-75cc8d5cb7-5wrts\" (UID: \"b406c2b2-6add-4e70-bdaa-c358471c2144\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5wrts"
Jul 29 20:53:16.272637 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:16.272614    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5wrts"]
Jul 29 20:53:16.366791 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:16.366763    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk2n2\" (UniqueName: \"kubernetes.io/projected/b406c2b2-6add-4e70-bdaa-c358471c2144-kube-api-access-hk2n2\") pod \"authorino-operator-75cc8d5cb7-5wrts\" (UID: \"b406c2b2-6add-4e70-bdaa-c358471c2144\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5wrts"
Jul 29 20:53:16.379776 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:16.379752    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk2n2\" (UniqueName: \"kubernetes.io/projected/b406c2b2-6add-4e70-bdaa-c358471c2144-kube-api-access-hk2n2\") pod \"authorino-operator-75cc8d5cb7-5wrts\" (UID: \"b406c2b2-6add-4e70-bdaa-c358471c2144\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5wrts"
Jul 29 20:53:16.566181 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:16.566089    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5wrts"
Jul 29 20:53:16.686978 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:16.686955    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5wrts"]
Jul 29 20:53:16.689548 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:53:16.689524    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb406c2b2_6add_4e70_bdaa_c358471c2144.slice/crio-4b9b0a6db4232d967e086c318ba10030a6373e642057ff4b127367e1b0846e57 WatchSource:0}: Error finding container 4b9b0a6db4232d967e086c318ba10030a6373e642057ff4b127367e1b0846e57: Status 404 returned error can't find the container with id 4b9b0a6db4232d967e086c318ba10030a6373e642057ff4b127367e1b0846e57
Jul 29 20:53:17.534652 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:17.534614    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5wrts" event={"ID":"b406c2b2-6add-4e70-bdaa-c358471c2144","Type":"ContainerStarted","Data":"4b9b0a6db4232d967e086c318ba10030a6373e642057ff4b127367e1b0846e57"}
Jul 29 20:53:19.544936 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:19.544899    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5wrts" event={"ID":"b406c2b2-6add-4e70-bdaa-c358471c2144","Type":"ContainerStarted","Data":"cf7c7cac7afab504eaaaff664863ba3899c223e7e3cf7bbf5c685d882542db30"}
Jul 29 20:53:19.545345 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:19.545021    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5wrts"
Jul 29 20:53:19.565459 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:19.565405    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5wrts" podStartSLOduration=1.328348673 podStartE2EDuration="3.565393005s" totalImagesPullingTime="2.237044332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:53:19.563660465 +0000 UTC m=+415.159952149" watchObservedRunningTime="2026-07-29 20:53:19.565393005 +0000 UTC m=+415.161684742"
Jul 29 20:53:30.551416 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:30.551385    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5wrts"
Jul 29 20:53:32.516040 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:32.516007    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x5mwx"]
Jul 29 20:53:32.519246 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:32.519228    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x5mwx"
Jul 29 20:53:32.521103 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:32.521080    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-s286g\""
Jul 29 20:53:32.534979 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:32.534949    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x5mwx"]
Jul 29 20:53:32.585255 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:32.585219    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dv8h\" (UniqueName: \"kubernetes.io/projected/6388a4d7-6919-407d-9dc4-c41a357e86af-kube-api-access-4dv8h\") pod \"limitador-operator-controller-manager-859d7b78bd-x5mwx\" (UID: \"6388a4d7-6919-407d-9dc4-c41a357e86af\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x5mwx"
Jul 29 20:53:32.686654 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:32.686620    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dv8h\" (UniqueName: \"kubernetes.io/projected/6388a4d7-6919-407d-9dc4-c41a357e86af-kube-api-access-4dv8h\") pod \"limitador-operator-controller-manager-859d7b78bd-x5mwx\" (UID: \"6388a4d7-6919-407d-9dc4-c41a357e86af\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x5mwx"
Jul 29 20:53:32.694333 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:32.694300    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dv8h\" (UniqueName: \"kubernetes.io/projected/6388a4d7-6919-407d-9dc4-c41a357e86af-kube-api-access-4dv8h\") pod \"limitador-operator-controller-manager-859d7b78bd-x5mwx\" (UID: \"6388a4d7-6919-407d-9dc4-c41a357e86af\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x5mwx"
Jul 29 20:53:32.829384 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:32.829303    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x5mwx"
Jul 29 20:53:32.956771 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:32.956740    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x5mwx"]
Jul 29 20:53:33.602403 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:33.602367    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x5mwx" event={"ID":"6388a4d7-6919-407d-9dc4-c41a357e86af","Type":"ContainerStarted","Data":"ea11c5f5381d17dea6a8ab17fa24cee1e2fd7c4574d8df808132dedec144e7a0"}
Jul 29 20:53:34.607997 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:34.607965    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x5mwx" event={"ID":"6388a4d7-6919-407d-9dc4-c41a357e86af","Type":"ContainerStarted","Data":"b070b5827581a54e33d95ea77d1702c6fbb9aea7ca1fb0ebc6ef8914a826ea58"}
Jul 29 20:53:34.608398 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:34.608032    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x5mwx"
Jul 29 20:53:34.623347 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:34.623297    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x5mwx" podStartSLOduration=1.13480533 podStartE2EDuration="2.623281413s" totalImagesPullingTime="1.488476083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:53:34.622110288 +0000 UTC m=+430.218401993" watchObservedRunningTime="2026-07-29 20:53:34.623281413 +0000 UTC m=+430.219573096"
Jul 29 20:53:45.619585 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:53:45.619558    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x5mwx"
Jul 29 20:54:07.860892 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:07.860858    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-84w2s"]
Jul 29 20:54:07.867126 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:07.867102    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-84w2s"
Jul 29 20:54:07.868895 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:07.868871    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hbtqk\""
Jul 29 20:54:07.871533 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:07.871509    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-84w2s"]
Jul 29 20:54:07.968445 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:07.968411    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ct4\" (UniqueName: \"kubernetes.io/projected/2b9bc998-86a1-4b3e-b767-442a75436c83-kube-api-access-h7ct4\") pod \"authorino-58cd955675-84w2s\" (UID: \"2b9bc998-86a1-4b3e-b767-442a75436c83\") " pod="kuadrant-system/authorino-58cd955675-84w2s"
Jul 29 20:54:08.030077 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.030043    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-grqrj"]
Jul 29 20:54:08.033284 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.033267    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-grqrj"
Jul 29 20:54:08.041900 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.040231    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-grqrj"]
Jul 29 20:54:08.068998 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.068958    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ct4\" (UniqueName: \"kubernetes.io/projected/2b9bc998-86a1-4b3e-b767-442a75436c83-kube-api-access-h7ct4\") pod \"authorino-58cd955675-84w2s\" (UID: \"2b9bc998-86a1-4b3e-b767-442a75436c83\") " pod="kuadrant-system/authorino-58cd955675-84w2s"
Jul 29 20:54:08.076604 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.076570    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ct4\" (UniqueName: \"kubernetes.io/projected/2b9bc998-86a1-4b3e-b767-442a75436c83-kube-api-access-h7ct4\") pod \"authorino-58cd955675-84w2s\" (UID: \"2b9bc998-86a1-4b3e-b767-442a75436c83\") " pod="kuadrant-system/authorino-58cd955675-84w2s"
Jul 29 20:54:08.169956 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.169864    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rs8j\" (UniqueName: \"kubernetes.io/projected/d911af23-6c59-4a34-987c-78fc6d65ac59-kube-api-access-5rs8j\") pod \"authorino-75d556bd77-grqrj\" (UID: \"d911af23-6c59-4a34-987c-78fc6d65ac59\") " pod="kuadrant-system/authorino-75d556bd77-grqrj"
Jul 29 20:54:08.176774 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.176752    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-84w2s"
Jul 29 20:54:08.270711 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.270681    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rs8j\" (UniqueName: \"kubernetes.io/projected/d911af23-6c59-4a34-987c-78fc6d65ac59-kube-api-access-5rs8j\") pod \"authorino-75d556bd77-grqrj\" (UID: \"d911af23-6c59-4a34-987c-78fc6d65ac59\") " pod="kuadrant-system/authorino-75d556bd77-grqrj"
Jul 29 20:54:08.278441 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.278413    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rs8j\" (UniqueName: \"kubernetes.io/projected/d911af23-6c59-4a34-987c-78fc6d65ac59-kube-api-access-5rs8j\") pod \"authorino-75d556bd77-grqrj\" (UID: \"d911af23-6c59-4a34-987c-78fc6d65ac59\") " pod="kuadrant-system/authorino-75d556bd77-grqrj"
Jul 29 20:54:08.306136 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.306104    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-84w2s"]
Jul 29 20:54:08.308593 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:54:08.308569    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b9bc998_86a1_4b3e_b767_442a75436c83.slice/crio-bd80e5d4cf07d912b6eb13c4699679cf55eb993a1bb0bedc94821769638e73b6 WatchSource:0}: Error finding container bd80e5d4cf07d912b6eb13c4699679cf55eb993a1bb0bedc94821769638e73b6: Status 404 returned error can't find the container with id bd80e5d4cf07d912b6eb13c4699679cf55eb993a1bb0bedc94821769638e73b6
Jul 29 20:54:08.342836 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.342800    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-grqrj"
Jul 29 20:54:08.470942 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.470919    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-grqrj"]
Jul 29 20:54:08.473287 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:54:08.473262    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd911af23_6c59_4a34_987c_78fc6d65ac59.slice/crio-7df3052a49fa58f6b801cdb4bf917b7b58738ae00a21b9c14260cf64960186ff WatchSource:0}: Error finding container 7df3052a49fa58f6b801cdb4bf917b7b58738ae00a21b9c14260cf64960186ff: Status 404 returned error can't find the container with id 7df3052a49fa58f6b801cdb4bf917b7b58738ae00a21b9c14260cf64960186ff
Jul 29 20:54:08.752902 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.752822    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-grqrj" event={"ID":"d911af23-6c59-4a34-987c-78fc6d65ac59","Type":"ContainerStarted","Data":"7df3052a49fa58f6b801cdb4bf917b7b58738ae00a21b9c14260cf64960186ff"}
Jul 29 20:54:08.753888 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:08.753867    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-84w2s" event={"ID":"2b9bc998-86a1-4b3e-b767-442a75436c83","Type":"ContainerStarted","Data":"bd80e5d4cf07d912b6eb13c4699679cf55eb993a1bb0bedc94821769638e73b6"}
Jul 29 20:54:11.770358 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:11.770325    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-grqrj" event={"ID":"d911af23-6c59-4a34-987c-78fc6d65ac59","Type":"ContainerStarted","Data":"0e453b2360b251d5ba08fdb23e721b6d6aa0814b411f3a8923aeffde11f99c32"}
Jul 29 20:54:11.771536 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:11.771507    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-84w2s" event={"ID":"2b9bc998-86a1-4b3e-b767-442a75436c83","Type":"ContainerStarted","Data":"60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15"}
Jul 29 20:54:11.785062 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:11.785019    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-grqrj" podStartSLOduration=0.561527083 podStartE2EDuration="3.785000094s" totalImagesPullingTime="3.223473011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:54:11.784024194 +0000 UTC m=+467.380315879" watchObservedRunningTime="2026-07-29 20:54:11.785000094 +0000 UTC m=+467.381291780"
Jul 29 20:54:11.800078 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:11.798114    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-84w2s" podStartSLOduration=1.420241888 podStartE2EDuration="4.798087093s" totalImagesPullingTime="3.377845205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:54:11.797636086 +0000 UTC m=+467.393927771" watchObservedRunningTime="2026-07-29 20:54:11.798087093 +0000 UTC m=+467.394378778"
Jul 29 20:54:11.821967 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:11.821878    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-84w2s"]
Jul 29 20:54:13.779883 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:13.779842    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-84w2s" podUID="2b9bc998-86a1-4b3e-b767-442a75436c83" containerName="authorino" containerID="cri-o://60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15" gracePeriod=30
Jul 29 20:54:13.802600 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:54:13.802570    2584 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b9bc998_86a1_4b3e_b767_442a75436c83.slice/crio-60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15.scope\": RecentStats: unable to find data in memory cache]"
Jul 29 20:54:14.019379 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.019356    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-84w2s"
Jul 29 20:54:14.125294 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.125268    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7ct4\" (UniqueName: \"kubernetes.io/projected/2b9bc998-86a1-4b3e-b767-442a75436c83-kube-api-access-h7ct4\") pod \"2b9bc998-86a1-4b3e-b767-442a75436c83\" (UID: \"2b9bc998-86a1-4b3e-b767-442a75436c83\") "
Jul 29 20:54:14.127402 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.127375    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9bc998-86a1-4b3e-b767-442a75436c83-kube-api-access-h7ct4" (OuterVolumeSpecName: "kube-api-access-h7ct4") pod "2b9bc998-86a1-4b3e-b767-442a75436c83" (UID: "2b9bc998-86a1-4b3e-b767-442a75436c83"). InnerVolumeSpecName "kube-api-access-h7ct4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:54:14.225913 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.225881    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7ct4\" (UniqueName: \"kubernetes.io/projected/2b9bc998-86a1-4b3e-b767-442a75436c83-kube-api-access-h7ct4\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:54:14.784947 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.784911    2584 generic.go:356] "Generic (PLEG): container finished" podID="2b9bc998-86a1-4b3e-b767-442a75436c83" containerID="60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15" exitCode=0
Jul 29 20:54:14.785359 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.784957    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-84w2s"
Jul 29 20:54:14.785359 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.784960    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-84w2s" event={"ID":"2b9bc998-86a1-4b3e-b767-442a75436c83","Type":"ContainerDied","Data":"60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15"}
Jul 29 20:54:14.785359 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.784998    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-84w2s" event={"ID":"2b9bc998-86a1-4b3e-b767-442a75436c83","Type":"ContainerDied","Data":"bd80e5d4cf07d912b6eb13c4699679cf55eb993a1bb0bedc94821769638e73b6"}
Jul 29 20:54:14.785359 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.785034    2584 scope.go:117] "RemoveContainer" containerID="60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15"
Jul 29 20:54:14.793595 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.793569    2584 scope.go:117] "RemoveContainer" containerID="60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15"
Jul 29 20:54:14.793849 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:54:14.793825    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15\": container with ID starting with 60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15 not found: ID does not exist" containerID="60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15"
Jul 29 20:54:14.793921 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.793855    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15"} err="failed to get container status \"60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15\": rpc error: code = NotFound desc = could not find container \"60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15\": container with ID starting with 60ccb484bb04fe43b96587dcc60cd16737e98be4d3a80c4e893fa0b32859aa15 not found: ID does not exist"
Jul 29 20:54:14.803827 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.803807    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-84w2s"]
Jul 29 20:54:14.809037 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.809017    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-84w2s"]
Jul 29 20:54:14.864895 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:14.864861    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b9bc998-86a1-4b3e-b767-442a75436c83" path="/var/lib/kubelet/pods/2b9bc998-86a1-4b3e-b767-442a75436c83/volumes"
Jul 29 20:54:40.858743 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:40.858704    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kqh5s"]
Jul 29 20:54:40.859334 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:40.859212    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b9bc998-86a1-4b3e-b767-442a75436c83" containerName="authorino"
Jul 29 20:54:40.859334 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:40.859233    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b9bc998-86a1-4b3e-b767-442a75436c83" containerName="authorino"
Jul 29 20:54:40.859334 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:40.859321    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b9bc998-86a1-4b3e-b767-442a75436c83" containerName="authorino"
Jul 29 20:54:40.864839 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:40.864812    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kqh5s"
Jul 29 20:54:40.872044 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:40.872019    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kqh5s"]
Jul 29 20:54:40.940170 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:40.940110    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxqvl\" (UniqueName: \"kubernetes.io/projected/82a2f61e-d7d6-49aa-bf62-891a7df9da91-kube-api-access-nxqvl\") pod \"authorino-5d4b67c446-kqh5s\" (UID: \"82a2f61e-d7d6-49aa-bf62-891a7df9da91\") " pod="kuadrant-system/authorino-5d4b67c446-kqh5s"
Jul 29 20:54:41.041156 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.041113    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqvl\" (UniqueName: \"kubernetes.io/projected/82a2f61e-d7d6-49aa-bf62-891a7df9da91-kube-api-access-nxqvl\") pod \"authorino-5d4b67c446-kqh5s\" (UID: \"82a2f61e-d7d6-49aa-bf62-891a7df9da91\") " pod="kuadrant-system/authorino-5d4b67c446-kqh5s"
Jul 29 20:54:41.049606 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.049571    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqvl\" (UniqueName: \"kubernetes.io/projected/82a2f61e-d7d6-49aa-bf62-891a7df9da91-kube-api-access-nxqvl\") pod \"authorino-5d4b67c446-kqh5s\" (UID: \"82a2f61e-d7d6-49aa-bf62-891a7df9da91\") " pod="kuadrant-system/authorino-5d4b67c446-kqh5s"
Jul 29 20:54:41.054650 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.054627    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kqh5s"]
Jul 29 20:54:41.054868 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.054855    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kqh5s"
Jul 29 20:54:41.081757 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.081728    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-kd28j"]
Jul 29 20:54:41.092939 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.092912    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-kd28j"
Jul 29 20:54:41.093246 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.093222    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-kd28j"]
Jul 29 20:54:41.094920 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.094897    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 29 20:54:41.116614 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.116583    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-kd28j"]
Jul 29 20:54:41.116958 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:54:41.116931    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-8btxr tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-8btxr tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-kd28j" podUID="cb8fa0c9-c35d-4539-a48b-ff669e78e8b6"
Jul 29 20:54:41.141705 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.141611    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8btxr\" (UniqueName: \"kubernetes.io/projected/cb8fa0c9-c35d-4539-a48b-ff669e78e8b6-kube-api-access-8btxr\") pod \"authorino-76877994-kd28j\" (UID: \"cb8fa0c9-c35d-4539-a48b-ff669e78e8b6\") " pod="kuadrant-system/authorino-76877994-kd28j"
Jul 29 20:54:41.141705 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.141678    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb8fa0c9-c35d-4539-a48b-ff669e78e8b6-tls-cert\") pod \"authorino-76877994-kd28j\" (UID: \"cb8fa0c9-c35d-4539-a48b-ff669e78e8b6\") " pod="kuadrant-system/authorino-76877994-kd28j"
Jul 29 20:54:41.143096 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.143071    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b4d8f9b8f-lpqt8"]
Jul 29 20:54:41.146883 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.146863    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8"
Jul 29 20:54:41.154847 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.154254    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b4d8f9b8f-lpqt8"]
Jul 29 20:54:41.192827 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.192799    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kqh5s"]
Jul 29 20:54:41.195901 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:54:41.195877    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82a2f61e_d7d6_49aa_bf62_891a7df9da91.slice/crio-131d307c61fb51c4f2ac27229e6e6dd6aedfd9dbd928fe6894608e7befb0e495 WatchSource:0}: Error finding container 131d307c61fb51c4f2ac27229e6e6dd6aedfd9dbd928fe6894608e7befb0e495: Status 404 returned error can't find the container with id 131d307c61fb51c4f2ac27229e6e6dd6aedfd9dbd928fe6894608e7befb0e495
Jul 29 20:54:41.242175 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.242119    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkhw8\" (UniqueName: \"kubernetes.io/projected/e2061533-7062-451f-80e1-78237e54c19b-kube-api-access-lkhw8\") pod \"authorino-b4d8f9b8f-lpqt8\" (UID: \"e2061533-7062-451f-80e1-78237e54c19b\") " pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8"
Jul 29 20:54:41.242332 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.242216    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2061533-7062-451f-80e1-78237e54c19b-tls-cert\") pod \"authorino-b4d8f9b8f-lpqt8\" (UID: \"e2061533-7062-451f-80e1-78237e54c19b\") " pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8"
Jul 29 20:54:41.242332 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.242253    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8btxr\" (UniqueName: \"kubernetes.io/projected/cb8fa0c9-c35d-4539-a48b-ff669e78e8b6-kube-api-access-8btxr\") pod \"authorino-76877994-kd28j\" (UID: \"cb8fa0c9-c35d-4539-a48b-ff669e78e8b6\") " pod="kuadrant-system/authorino-76877994-kd28j"
Jul 29 20:54:41.242332 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.242285    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb8fa0c9-c35d-4539-a48b-ff669e78e8b6-tls-cert\") pod \"authorino-76877994-kd28j\" (UID: \"cb8fa0c9-c35d-4539-a48b-ff669e78e8b6\") " pod="kuadrant-system/authorino-76877994-kd28j"
Jul 29 20:54:41.244655 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.244632    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb8fa0c9-c35d-4539-a48b-ff669e78e8b6-tls-cert\") pod \"authorino-76877994-kd28j\" (UID: \"cb8fa0c9-c35d-4539-a48b-ff669e78e8b6\") " pod="kuadrant-system/authorino-76877994-kd28j"
Jul 29 20:54:41.249069 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.249048    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8btxr\" (UniqueName: \"kubernetes.io/projected/cb8fa0c9-c35d-4539-a48b-ff669e78e8b6-kube-api-access-8btxr\") pod \"authorino-76877994-kd28j\" (UID: \"cb8fa0c9-c35d-4539-a48b-ff669e78e8b6\") " pod="kuadrant-system/authorino-76877994-kd28j"
Jul 29 20:54:41.345176 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.343478    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2061533-7062-451f-80e1-78237e54c19b-tls-cert\") pod \"authorino-b4d8f9b8f-lpqt8\" (UID: \"e2061533-7062-451f-80e1-78237e54c19b\") " pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8"
Jul 29 20:54:41.345176 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.343638    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkhw8\" (UniqueName: \"kubernetes.io/projected/e2061533-7062-451f-80e1-78237e54c19b-kube-api-access-lkhw8\") pod \"authorino-b4d8f9b8f-lpqt8\" (UID: \"e2061533-7062-451f-80e1-78237e54c19b\") " pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8"
Jul 29 20:54:41.348388 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.347121    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2061533-7062-451f-80e1-78237e54c19b-tls-cert\") pod \"authorino-b4d8f9b8f-lpqt8\" (UID: \"e2061533-7062-451f-80e1-78237e54c19b\") " pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8"
Jul 29 20:54:41.351114 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.351092    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkhw8\" (UniqueName: \"kubernetes.io/projected/e2061533-7062-451f-80e1-78237e54c19b-kube-api-access-lkhw8\") pod \"authorino-b4d8f9b8f-lpqt8\" (UID: \"e2061533-7062-451f-80e1-78237e54c19b\") " pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8"
Jul 29 20:54:41.458586 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.458505    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8"
Jul 29 20:54:41.583656 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.583632    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b4d8f9b8f-lpqt8"]
Jul 29 20:54:41.587018 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:54:41.586988    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2061533_7062_451f_80e1_78237e54c19b.slice/crio-c9a69e3c83657d0b2d3988c2c6396cd6f5388217ac1f6b7394b0007e79e3ee04 WatchSource:0}: Error finding container c9a69e3c83657d0b2d3988c2c6396cd6f5388217ac1f6b7394b0007e79e3ee04: Status 404 returned error can't find the container with id c9a69e3c83657d0b2d3988c2c6396cd6f5388217ac1f6b7394b0007e79e3ee04
Jul 29 20:54:41.895263 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.895230    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-kqh5s" event={"ID":"82a2f61e-d7d6-49aa-bf62-891a7df9da91","Type":"ContainerStarted","Data":"f547c003ed683603634e5c96cfeafcbd089665ffcfef04a57814e4fbaef70471"}
Jul 29 20:54:41.895263 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.895251    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-kqh5s" podUID="82a2f61e-d7d6-49aa-bf62-891a7df9da91" containerName="authorino" containerID="cri-o://f547c003ed683603634e5c96cfeafcbd089665ffcfef04a57814e4fbaef70471" gracePeriod=30
Jul 29 20:54:41.895761 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.895272    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-kqh5s" event={"ID":"82a2f61e-d7d6-49aa-bf62-891a7df9da91","Type":"ContainerStarted","Data":"131d307c61fb51c4f2ac27229e6e6dd6aedfd9dbd928fe6894608e7befb0e495"}
Jul 29 20:54:41.896433 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.896414    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8" event={"ID":"e2061533-7062-451f-80e1-78237e54c19b","Type":"ContainerStarted","Data":"c9a69e3c83657d0b2d3988c2c6396cd6f5388217ac1f6b7394b0007e79e3ee04"}
Jul 29 20:54:41.896520 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.896444    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-kd28j"
Jul 29 20:54:41.903125 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.903105    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-kd28j"
Jul 29 20:54:41.908726 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.908682    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-kqh5s" podStartSLOduration=1.533156166 podStartE2EDuration="1.908668602s" totalImagesPullingTime="375.512436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:54:41.907706833 +0000 UTC m=+497.503998518" watchObservedRunningTime="2026-07-29 20:54:41.908668602 +0000 UTC m=+497.504960287"
Jul 29 20:54:41.949081 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.949053    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb8fa0c9-c35d-4539-a48b-ff669e78e8b6-tls-cert\") pod \"cb8fa0c9-c35d-4539-a48b-ff669e78e8b6\" (UID: \"cb8fa0c9-c35d-4539-a48b-ff669e78e8b6\") "
Jul 29 20:54:41.949238 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.949107    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8btxr\" (UniqueName: \"kubernetes.io/projected/cb8fa0c9-c35d-4539-a48b-ff669e78e8b6-kube-api-access-8btxr\") pod \"cb8fa0c9-c35d-4539-a48b-ff669e78e8b6\" (UID: \"cb8fa0c9-c35d-4539-a48b-ff669e78e8b6\") "
Jul 29 20:54:41.951091 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.951059    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb8fa0c9-c35d-4539-a48b-ff669e78e8b6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "cb8fa0c9-c35d-4539-a48b-ff669e78e8b6" (UID: "cb8fa0c9-c35d-4539-a48b-ff669e78e8b6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:54:41.951091 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:41.951081    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8fa0c9-c35d-4539-a48b-ff669e78e8b6-kube-api-access-8btxr" (OuterVolumeSpecName: "kube-api-access-8btxr") pod "cb8fa0c9-c35d-4539-a48b-ff669e78e8b6" (UID: "cb8fa0c9-c35d-4539-a48b-ff669e78e8b6"). InnerVolumeSpecName "kube-api-access-8btxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:54:42.050323 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.050290    2584 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb8fa0c9-c35d-4539-a48b-ff669e78e8b6-tls-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:54:42.050323 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.050326    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8btxr\" (UniqueName: \"kubernetes.io/projected/cb8fa0c9-c35d-4539-a48b-ff669e78e8b6-kube-api-access-8btxr\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:54:42.160650 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.160622    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kqh5s"
Jul 29 20:54:42.252053 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.252017    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxqvl\" (UniqueName: \"kubernetes.io/projected/82a2f61e-d7d6-49aa-bf62-891a7df9da91-kube-api-access-nxqvl\") pod \"82a2f61e-d7d6-49aa-bf62-891a7df9da91\" (UID: \"82a2f61e-d7d6-49aa-bf62-891a7df9da91\") "
Jul 29 20:54:42.254040 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.254012    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a2f61e-d7d6-49aa-bf62-891a7df9da91-kube-api-access-nxqvl" (OuterVolumeSpecName: "kube-api-access-nxqvl") pod "82a2f61e-d7d6-49aa-bf62-891a7df9da91" (UID: "82a2f61e-d7d6-49aa-bf62-891a7df9da91"). InnerVolumeSpecName "kube-api-access-nxqvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:54:42.353239 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.353201    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxqvl\" (UniqueName: \"kubernetes.io/projected/82a2f61e-d7d6-49aa-bf62-891a7df9da91-kube-api-access-nxqvl\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:54:42.901355 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.901321    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8" event={"ID":"e2061533-7062-451f-80e1-78237e54c19b","Type":"ContainerStarted","Data":"96ead089594968bdd524689593ab355be4b43a944c243987a1c36fd9a2974c95"}
Jul 29 20:54:42.902519 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.902484    2584 generic.go:356] "Generic (PLEG): container finished" podID="82a2f61e-d7d6-49aa-bf62-891a7df9da91" containerID="f547c003ed683603634e5c96cfeafcbd089665ffcfef04a57814e4fbaef70471" exitCode=0
Jul 29 20:54:42.902519 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.902514    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kqh5s"
Jul 29 20:54:42.902662 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.902542    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-kqh5s" event={"ID":"82a2f61e-d7d6-49aa-bf62-891a7df9da91","Type":"ContainerDied","Data":"f547c003ed683603634e5c96cfeafcbd089665ffcfef04a57814e4fbaef70471"}
Jul 29 20:54:42.902662 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.902567    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-kqh5s" event={"ID":"82a2f61e-d7d6-49aa-bf62-891a7df9da91","Type":"ContainerDied","Data":"131d307c61fb51c4f2ac27229e6e6dd6aedfd9dbd928fe6894608e7befb0e495"}
Jul 29 20:54:42.902662 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.902584    2584 scope.go:117] "RemoveContainer" containerID="f547c003ed683603634e5c96cfeafcbd089665ffcfef04a57814e4fbaef70471"
Jul 29 20:54:42.902804 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.902690    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-kd28j"
Jul 29 20:54:42.910243 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.910223    2584 scope.go:117] "RemoveContainer" containerID="f547c003ed683603634e5c96cfeafcbd089665ffcfef04a57814e4fbaef70471"
Jul 29 20:54:42.910492 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:54:42.910473    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f547c003ed683603634e5c96cfeafcbd089665ffcfef04a57814e4fbaef70471\": container with ID starting with f547c003ed683603634e5c96cfeafcbd089665ffcfef04a57814e4fbaef70471 not found: ID does not exist" containerID="f547c003ed683603634e5c96cfeafcbd089665ffcfef04a57814e4fbaef70471"
Jul 29 20:54:42.910549 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.910503    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f547c003ed683603634e5c96cfeafcbd089665ffcfef04a57814e4fbaef70471"} err="failed to get container status \"f547c003ed683603634e5c96cfeafcbd089665ffcfef04a57814e4fbaef70471\": rpc error: code = NotFound desc = could not find container \"f547c003ed683603634e5c96cfeafcbd089665ffcfef04a57814e4fbaef70471\": container with ID starting with f547c003ed683603634e5c96cfeafcbd089665ffcfef04a57814e4fbaef70471 not found: ID does not exist"
Jul 29 20:54:42.915839 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.915797    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8" podStartSLOduration=1.522535768 podStartE2EDuration="1.915784671s" totalImagesPullingTime="393.248903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:54:42.914775587 +0000 UTC m=+498.511067273" watchObservedRunningTime="2026-07-29 20:54:42.915784671 +0000 UTC m=+498.512076354"
Jul 29 20:54:42.936690 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.936656    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-kd28j"]
Jul 29 20:54:42.941661 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.941638    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-grqrj"]
Jul 29 20:54:42.941810 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.941791    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-grqrj" podUID="d911af23-6c59-4a34-987c-78fc6d65ac59" containerName="authorino" containerID="cri-o://0e453b2360b251d5ba08fdb23e721b6d6aa0814b411f3a8923aeffde11f99c32" gracePeriod=30
Jul 29 20:54:42.944195 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.944173    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-kd28j"]
Jul 29 20:54:42.957862 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.957835    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kqh5s"]
Jul 29 20:54:42.964968 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:42.964940    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kqh5s"]
Jul 29 20:54:43.193939 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:43.193915    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-grqrj"
Jul 29 20:54:43.260468 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:43.260438    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rs8j\" (UniqueName: \"kubernetes.io/projected/d911af23-6c59-4a34-987c-78fc6d65ac59-kube-api-access-5rs8j\") pod \"d911af23-6c59-4a34-987c-78fc6d65ac59\" (UID: \"d911af23-6c59-4a34-987c-78fc6d65ac59\") "
Jul 29 20:54:43.262525 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:43.262491    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d911af23-6c59-4a34-987c-78fc6d65ac59-kube-api-access-5rs8j" (OuterVolumeSpecName: "kube-api-access-5rs8j") pod "d911af23-6c59-4a34-987c-78fc6d65ac59" (UID: "d911af23-6c59-4a34-987c-78fc6d65ac59"). InnerVolumeSpecName "kube-api-access-5rs8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:54:43.361807 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:43.361769    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rs8j\" (UniqueName: \"kubernetes.io/projected/d911af23-6c59-4a34-987c-78fc6d65ac59-kube-api-access-5rs8j\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:54:43.908242 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:43.908211    2584 generic.go:356] "Generic (PLEG): container finished" podID="d911af23-6c59-4a34-987c-78fc6d65ac59" containerID="0e453b2360b251d5ba08fdb23e721b6d6aa0814b411f3a8923aeffde11f99c32" exitCode=0
Jul 29 20:54:43.908657 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:43.908250    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-grqrj"
Jul 29 20:54:43.908657 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:43.908289    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-grqrj" event={"ID":"d911af23-6c59-4a34-987c-78fc6d65ac59","Type":"ContainerDied","Data":"0e453b2360b251d5ba08fdb23e721b6d6aa0814b411f3a8923aeffde11f99c32"}
Jul 29 20:54:43.908657 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:43.908325    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-grqrj" event={"ID":"d911af23-6c59-4a34-987c-78fc6d65ac59","Type":"ContainerDied","Data":"7df3052a49fa58f6b801cdb4bf917b7b58738ae00a21b9c14260cf64960186ff"}
Jul 29 20:54:43.908657 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:43.908343    2584 scope.go:117] "RemoveContainer" containerID="0e453b2360b251d5ba08fdb23e721b6d6aa0814b411f3a8923aeffde11f99c32"
Jul 29 20:54:43.916246 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:43.916229    2584 scope.go:117] "RemoveContainer" containerID="0e453b2360b251d5ba08fdb23e721b6d6aa0814b411f3a8923aeffde11f99c32"
Jul 29 20:54:43.916477 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:54:43.916463    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e453b2360b251d5ba08fdb23e721b6d6aa0814b411f3a8923aeffde11f99c32\": container with ID starting with 0e453b2360b251d5ba08fdb23e721b6d6aa0814b411f3a8923aeffde11f99c32 not found: ID does not exist" containerID="0e453b2360b251d5ba08fdb23e721b6d6aa0814b411f3a8923aeffde11f99c32"
Jul 29 20:54:43.916520 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:43.916486    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e453b2360b251d5ba08fdb23e721b6d6aa0814b411f3a8923aeffde11f99c32"} err="failed to get container status \"0e453b2360b251d5ba08fdb23e721b6d6aa0814b411f3a8923aeffde11f99c32\": rpc error: code = NotFound desc = could not find container \"0e453b2360b251d5ba08fdb23e721b6d6aa0814b411f3a8923aeffde11f99c32\": container with ID starting with 0e453b2360b251d5ba08fdb23e721b6d6aa0814b411f3a8923aeffde11f99c32 not found: ID does not exist"
Jul 29 20:54:43.926245 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:43.926225    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-grqrj"]
Jul 29 20:54:43.929840 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:43.929820    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-grqrj"]
Jul 29 20:54:44.864218 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:44.864181    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82a2f61e-d7d6-49aa-bf62-891a7df9da91" path="/var/lib/kubelet/pods/82a2f61e-d7d6-49aa-bf62-891a7df9da91/volumes"
Jul 29 20:54:44.864541 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:44.864529    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb8fa0c9-c35d-4539-a48b-ff669e78e8b6" path="/var/lib/kubelet/pods/cb8fa0c9-c35d-4539-a48b-ff669e78e8b6/volumes"
Jul 29 20:54:44.864733 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:54:44.864722    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d911af23-6c59-4a34-987c-78fc6d65ac59" path="/var/lib/kubelet/pods/d911af23-6c59-4a34-987c-78fc6d65ac59/volumes"
Jul 29 20:56:33.450636 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.450605    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"]
Jul 29 20:56:33.451924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.450925    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a2f61e-d7d6-49aa-bf62-891a7df9da91" containerName="authorino"
Jul 29 20:56:33.451924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.450938    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a2f61e-d7d6-49aa-bf62-891a7df9da91" containerName="authorino"
Jul 29 20:56:33.451924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.450953    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d911af23-6c59-4a34-987c-78fc6d65ac59" containerName="authorino"
Jul 29 20:56:33.451924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.450958    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d911af23-6c59-4a34-987c-78fc6d65ac59" containerName="authorino"
Jul 29 20:56:33.451924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.451010    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d911af23-6c59-4a34-987c-78fc6d65ac59" containerName="authorino"
Jul 29 20:56:33.451924 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.451021    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a2f61e-d7d6-49aa-bf62-891a7df9da91" containerName="authorino"
Jul 29 20:56:33.452721 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.452703    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.454797 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.454778    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 20:56:33.455619 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.455596    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 29 20:56:33.455739 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.455620    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t86zw\""
Jul 29 20:56:33.455739 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.455691    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 20:56:33.464733 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.464713    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"]
Jul 29 20:56:33.545751 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.545722    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsjf4\" (UniqueName: \"kubernetes.io/projected/b96d1ae2-829d-4164-a167-7c6ce09b109e-kube-api-access-hsjf4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.545876 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.545759    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96d1ae2-829d-4164-a167-7c6ce09b109e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.545876 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.545799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96d1ae2-829d-4164-a167-7c6ce09b109e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.545876 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.545854    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96d1ae2-829d-4164-a167-7c6ce09b109e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.545984 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.545877    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96d1ae2-829d-4164-a167-7c6ce09b109e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.545984 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.545896    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96d1ae2-829d-4164-a167-7c6ce09b109e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.646508 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.646479    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96d1ae2-829d-4164-a167-7c6ce09b109e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.646699 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.646515    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96d1ae2-829d-4164-a167-7c6ce09b109e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.646699 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.646553    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsjf4\" (UniqueName: \"kubernetes.io/projected/b96d1ae2-829d-4164-a167-7c6ce09b109e-kube-api-access-hsjf4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.646699 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.646572    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96d1ae2-829d-4164-a167-7c6ce09b109e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.646699 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.646608    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96d1ae2-829d-4164-a167-7c6ce09b109e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.646699 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.646655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96d1ae2-829d-4164-a167-7c6ce09b109e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.646961 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.646928    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96d1ae2-829d-4164-a167-7c6ce09b109e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.647036 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.646964    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96d1ae2-829d-4164-a167-7c6ce09b109e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.647081 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.647047    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96d1ae2-829d-4164-a167-7c6ce09b109e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.648865 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.648834    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96d1ae2-829d-4164-a167-7c6ce09b109e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.649134 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.649117    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96d1ae2-829d-4164-a167-7c6ce09b109e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.653858 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.653839    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsjf4\" (UniqueName: \"kubernetes.io/projected/b96d1ae2-829d-4164-a167-7c6ce09b109e-kube-api-access-hsjf4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gkmlq\" (UID: \"b96d1ae2-829d-4164-a167-7c6ce09b109e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.765316 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.765235    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:33.888488 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.888453    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"]
Jul 29 20:56:33.892183 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:56:33.892138    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96d1ae2_829d_4164_a167_7c6ce09b109e.slice/crio-21bf5cfbea511a4c24eb7a0226af812608d49ade5ddd5a62695b04930c73f3f1 WatchSource:0}: Error finding container 21bf5cfbea511a4c24eb7a0226af812608d49ade5ddd5a62695b04930c73f3f1: Status 404 returned error can't find the container with id 21bf5cfbea511a4c24eb7a0226af812608d49ade5ddd5a62695b04930c73f3f1
Jul 29 20:56:33.893727 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:33.893711    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:56:34.361874 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:34.361831    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq" event={"ID":"b96d1ae2-829d-4164-a167-7c6ce09b109e","Type":"ContainerStarted","Data":"21bf5cfbea511a4c24eb7a0226af812608d49ade5ddd5a62695b04930c73f3f1"}
Jul 29 20:56:35.368509 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:35.368473    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq" event={"ID":"b96d1ae2-829d-4164-a167-7c6ce09b109e","Type":"ContainerStarted","Data":"3c8d9bc420311a75365f48692e59abb88654aaab55703b77de9f1ef06e1f66be"}
Jul 29 20:56:35.384201 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:35.384135    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq" podStartSLOduration=1.396363269 podStartE2EDuration="2.384120124s" totalImagesPullingTime="987.756855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:35.383062391 +0000 UTC m=+610.979354096" watchObservedRunningTime="2026-07-29 20:56:35.384120124 +0000 UTC m=+610.980411808"
Jul 29 20:56:43.765344 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:43.765306    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:43.765748 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:43.765378    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:43.779519 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:43.779495    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:56:44.420074 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:56:44.420049    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gkmlq"
Jul 29 20:57:55.564098 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:55.564060    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54cd85c6d4-lpd8s"]
Jul 29 20:57:55.567260 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:55.567242    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54cd85c6d4-lpd8s"
Jul 29 20:57:55.574261 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:55.574235    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54cd85c6d4-lpd8s"]
Jul 29 20:57:55.603013 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:55.602986    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f405d7a-0815-4971-9df0-1d981d95da5f-tls-cert\") pod \"authorino-54cd85c6d4-lpd8s\" (UID: \"8f405d7a-0815-4971-9df0-1d981d95da5f\") " pod="kuadrant-system/authorino-54cd85c6d4-lpd8s"
Jul 29 20:57:55.603158 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:55.603039    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84kdr\" (UniqueName: \"kubernetes.io/projected/8f405d7a-0815-4971-9df0-1d981d95da5f-kube-api-access-84kdr\") pod \"authorino-54cd85c6d4-lpd8s\" (UID: \"8f405d7a-0815-4971-9df0-1d981d95da5f\") " pod="kuadrant-system/authorino-54cd85c6d4-lpd8s"
Jul 29 20:57:55.704059 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:55.704024    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f405d7a-0815-4971-9df0-1d981d95da5f-tls-cert\") pod \"authorino-54cd85c6d4-lpd8s\" (UID: \"8f405d7a-0815-4971-9df0-1d981d95da5f\") " pod="kuadrant-system/authorino-54cd85c6d4-lpd8s"
Jul 29 20:57:55.704244 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:55.704083    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84kdr\" (UniqueName: \"kubernetes.io/projected/8f405d7a-0815-4971-9df0-1d981d95da5f-kube-api-access-84kdr\") pod \"authorino-54cd85c6d4-lpd8s\" (UID: \"8f405d7a-0815-4971-9df0-1d981d95da5f\") " pod="kuadrant-system/authorino-54cd85c6d4-lpd8s"
Jul 29 20:57:55.706447 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:55.706420    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f405d7a-0815-4971-9df0-1d981d95da5f-tls-cert\") pod \"authorino-54cd85c6d4-lpd8s\" (UID: \"8f405d7a-0815-4971-9df0-1d981d95da5f\") " pod="kuadrant-system/authorino-54cd85c6d4-lpd8s"
Jul 29 20:57:55.710903 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:55.710880    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84kdr\" (UniqueName: \"kubernetes.io/projected/8f405d7a-0815-4971-9df0-1d981d95da5f-kube-api-access-84kdr\") pod \"authorino-54cd85c6d4-lpd8s\" (UID: \"8f405d7a-0815-4971-9df0-1d981d95da5f\") " pod="kuadrant-system/authorino-54cd85c6d4-lpd8s"
Jul 29 20:57:55.877287 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:55.877239    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54cd85c6d4-lpd8s"
Jul 29 20:57:55.999601 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:55.999505    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54cd85c6d4-lpd8s"]
Jul 29 20:57:56.002210 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:57:56.002181    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f405d7a_0815_4971_9df0_1d981d95da5f.slice/crio-f014a9b2354b38e7ce8ac80075d1b3603645f747a800dcb593bcc378a08e588c WatchSource:0}: Error finding container f014a9b2354b38e7ce8ac80075d1b3603645f747a800dcb593bcc378a08e588c: Status 404 returned error can't find the container with id f014a9b2354b38e7ce8ac80075d1b3603645f747a800dcb593bcc378a08e588c
Jul 29 20:57:56.702340 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:56.702306    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54cd85c6d4-lpd8s" event={"ID":"8f405d7a-0815-4971-9df0-1d981d95da5f","Type":"ContainerStarted","Data":"1bc20511e55db53425313e41b0ef3002aaacf337b20ea4f4a2d3baaef90383c5"}
Jul 29 20:57:56.702340 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:56.702349    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54cd85c6d4-lpd8s" event={"ID":"8f405d7a-0815-4971-9df0-1d981d95da5f","Type":"ContainerStarted","Data":"f014a9b2354b38e7ce8ac80075d1b3603645f747a800dcb593bcc378a08e588c"}
Jul 29 20:57:56.721279 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:56.721229    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-54cd85c6d4-lpd8s" podStartSLOduration=1.230919851 podStartE2EDuration="1.721210488s" totalImagesPullingTime="490.290637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:57:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:57:56.719632073 +0000 UTC m=+692.315923759" watchObservedRunningTime="2026-07-29 20:57:56.721210488 +0000 UTC m=+692.317502172"
Jul 29 20:57:56.750645 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:56.750558    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b4d8f9b8f-lpqt8"]
Jul 29 20:57:56.750804 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:56.750739    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8" podUID="e2061533-7062-451f-80e1-78237e54c19b" containerName="authorino" containerID="cri-o://96ead089594968bdd524689593ab355be4b43a944c243987a1c36fd9a2974c95" gracePeriod=30
Jul 29 20:57:57.019634 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.019606    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8"
Jul 29 20:57:57.115279 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.115239    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2061533-7062-451f-80e1-78237e54c19b-tls-cert\") pod \"e2061533-7062-451f-80e1-78237e54c19b\" (UID: \"e2061533-7062-451f-80e1-78237e54c19b\") "
Jul 29 20:57:57.115440 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.115300    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkhw8\" (UniqueName: \"kubernetes.io/projected/e2061533-7062-451f-80e1-78237e54c19b-kube-api-access-lkhw8\") pod \"e2061533-7062-451f-80e1-78237e54c19b\" (UID: \"e2061533-7062-451f-80e1-78237e54c19b\") "
Jul 29 20:57:57.117413 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.117379    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2061533-7062-451f-80e1-78237e54c19b-kube-api-access-lkhw8" (OuterVolumeSpecName: "kube-api-access-lkhw8") pod "e2061533-7062-451f-80e1-78237e54c19b" (UID: "e2061533-7062-451f-80e1-78237e54c19b"). InnerVolumeSpecName "kube-api-access-lkhw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:57:57.126923 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.126891    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2061533-7062-451f-80e1-78237e54c19b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e2061533-7062-451f-80e1-78237e54c19b" (UID: "e2061533-7062-451f-80e1-78237e54c19b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:57:57.216229 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.216195    2584 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2061533-7062-451f-80e1-78237e54c19b-tls-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:57:57.216229 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.216224    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkhw8\" (UniqueName: \"kubernetes.io/projected/e2061533-7062-451f-80e1-78237e54c19b-kube-api-access-lkhw8\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 20:57:57.708211 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.708171    2584 generic.go:356] "Generic (PLEG): container finished" podID="e2061533-7062-451f-80e1-78237e54c19b" containerID="96ead089594968bdd524689593ab355be4b43a944c243987a1c36fd9a2974c95" exitCode=0
Jul 29 20:57:57.708211 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.708206    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8"
Jul 29 20:57:57.708850 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.708250    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8" event={"ID":"e2061533-7062-451f-80e1-78237e54c19b","Type":"ContainerDied","Data":"96ead089594968bdd524689593ab355be4b43a944c243987a1c36fd9a2974c95"}
Jul 29 20:57:57.708850 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.708313    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b4d8f9b8f-lpqt8" event={"ID":"e2061533-7062-451f-80e1-78237e54c19b","Type":"ContainerDied","Data":"c9a69e3c83657d0b2d3988c2c6396cd6f5388217ac1f6b7394b0007e79e3ee04"}
Jul 29 20:57:57.708850 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.708344    2584 scope.go:117] "RemoveContainer" containerID="96ead089594968bdd524689593ab355be4b43a944c243987a1c36fd9a2974c95"
Jul 29 20:57:57.719835 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.719817    2584 scope.go:117] "RemoveContainer" containerID="96ead089594968bdd524689593ab355be4b43a944c243987a1c36fd9a2974c95"
Jul 29 20:57:57.720085 ip-10-0-129-194 kubenswrapper[2584]: E0729 20:57:57.720068    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ead089594968bdd524689593ab355be4b43a944c243987a1c36fd9a2974c95\": container with ID starting with 96ead089594968bdd524689593ab355be4b43a944c243987a1c36fd9a2974c95 not found: ID does not exist" containerID="96ead089594968bdd524689593ab355be4b43a944c243987a1c36fd9a2974c95"
Jul 29 20:57:57.720217 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.720097    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ead089594968bdd524689593ab355be4b43a944c243987a1c36fd9a2974c95"} err="failed to get container status \"96ead089594968bdd524689593ab355be4b43a944c243987a1c36fd9a2974c95\": rpc error: code = NotFound desc = could not find container \"96ead089594968bdd524689593ab355be4b43a944c243987a1c36fd9a2974c95\": container with ID starting with 96ead089594968bdd524689593ab355be4b43a944c243987a1c36fd9a2974c95 not found: ID does not exist"
Jul 29 20:57:57.732596 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.732572    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b4d8f9b8f-lpqt8"]
Jul 29 20:57:57.734874 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:57.734852    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b4d8f9b8f-lpqt8"]
Jul 29 20:57:58.864755 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:57:58.864725    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2061533-7062-451f-80e1-78237e54c19b" path="/var/lib/kubelet/pods/e2061533-7062-451f-80e1-78237e54c19b/volumes"
Jul 29 20:59:45.675216 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.675182    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c85847677-kgp5f"]
Jul 29 20:59:45.675659 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.675492    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2061533-7062-451f-80e1-78237e54c19b" containerName="authorino"
Jul 29 20:59:45.675659 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.675503    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2061533-7062-451f-80e1-78237e54c19b" containerName="authorino"
Jul 29 20:59:45.675659 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.675555    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2061533-7062-451f-80e1-78237e54c19b" containerName="authorino"
Jul 29 20:59:45.678323 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.678307    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c85847677-kgp5f"
Jul 29 20:59:45.680846 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.680827    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pvk7c\""
Jul 29 20:59:45.680846 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.680828    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 20:59:45.681058 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.680828    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 20:59:45.687861 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.687840    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c85847677-kgp5f"]
Jul 29 20:59:45.707577 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.707547    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b49a4bd-6383-43b5-bd36-147408502e8d-webhook-cert\") pod \"maas-controller-5c85847677-kgp5f\" (UID: \"8b49a4bd-6383-43b5-bd36-147408502e8d\") " pod="opendatahub/maas-controller-5c85847677-kgp5f"
Jul 29 20:59:45.707694 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.707598    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2ls4\" (UniqueName: \"kubernetes.io/projected/8b49a4bd-6383-43b5-bd36-147408502e8d-kube-api-access-w2ls4\") pod \"maas-controller-5c85847677-kgp5f\" (UID: \"8b49a4bd-6383-43b5-bd36-147408502e8d\") " pod="opendatahub/maas-controller-5c85847677-kgp5f"
Jul 29 20:59:45.808149 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.808126    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b49a4bd-6383-43b5-bd36-147408502e8d-webhook-cert\") pod \"maas-controller-5c85847677-kgp5f\" (UID: \"8b49a4bd-6383-43b5-bd36-147408502e8d\") " pod="opendatahub/maas-controller-5c85847677-kgp5f"
Jul 29 20:59:45.808300 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.808230    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ls4\" (UniqueName: \"kubernetes.io/projected/8b49a4bd-6383-43b5-bd36-147408502e8d-kube-api-access-w2ls4\") pod \"maas-controller-5c85847677-kgp5f\" (UID: \"8b49a4bd-6383-43b5-bd36-147408502e8d\") " pod="opendatahub/maas-controller-5c85847677-kgp5f"
Jul 29 20:59:45.810481 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.810460    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b49a4bd-6383-43b5-bd36-147408502e8d-webhook-cert\") pod \"maas-controller-5c85847677-kgp5f\" (UID: \"8b49a4bd-6383-43b5-bd36-147408502e8d\") " pod="opendatahub/maas-controller-5c85847677-kgp5f"
Jul 29 20:59:45.815582 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.815561    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ls4\" (UniqueName: \"kubernetes.io/projected/8b49a4bd-6383-43b5-bd36-147408502e8d-kube-api-access-w2ls4\") pod \"maas-controller-5c85847677-kgp5f\" (UID: \"8b49a4bd-6383-43b5-bd36-147408502e8d\") " pod="opendatahub/maas-controller-5c85847677-kgp5f"
Jul 29 20:59:45.988792 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:45.988709    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c85847677-kgp5f"
Jul 29 20:59:46.108803 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:46.108778    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c85847677-kgp5f"]
Jul 29 20:59:46.111201 ip-10-0-129-194 kubenswrapper[2584]: W0729 20:59:46.111173    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b49a4bd_6383_43b5_bd36_147408502e8d.slice/crio-9c6c8048fe59233c61ee9226ec9fbf6973df58521417019b96a2f89292b0d215 WatchSource:0}: Error finding container 9c6c8048fe59233c61ee9226ec9fbf6973df58521417019b96a2f89292b0d215: Status 404 returned error can't find the container with id 9c6c8048fe59233c61ee9226ec9fbf6973df58521417019b96a2f89292b0d215
Jul 29 20:59:46.154812 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:46.154786    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c85847677-kgp5f" event={"ID":"8b49a4bd-6383-43b5-bd36-147408502e8d","Type":"ContainerStarted","Data":"9c6c8048fe59233c61ee9226ec9fbf6973df58521417019b96a2f89292b0d215"}
Jul 29 20:59:49.169407 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:49.169365    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c85847677-kgp5f" event={"ID":"8b49a4bd-6383-43b5-bd36-147408502e8d","Type":"ContainerStarted","Data":"b93912cac5502fd74d01fe09578b29734bc9c0128fe00be3d11471ca25d41625"}
Jul 29 20:59:49.169766 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:49.169495    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c85847677-kgp5f"
Jul 29 20:59:49.185257 ip-10-0-129-194 kubenswrapper[2584]: I0729 20:59:49.185116    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c85847677-kgp5f" podStartSLOduration=2.046467138 podStartE2EDuration="4.185099864s" totalImagesPullingTime="2.138632726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:59:49.184396319 +0000 UTC m=+804.780688025" watchObservedRunningTime="2026-07-29 20:59:49.185099864 +0000 UTC m=+804.781391552"
Jul 29 21:00:00.139752 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:00.139718    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z"]
Jul 29 21:00:00.142980 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:00.142964    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z"
Jul 29 21:00:00.144920 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:00.144900    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\""
Jul 29 21:00:00.150685 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:00.150664    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z"]
Jul 29 21:00:00.153558 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:00.153540    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z"
Jul 29 21:00:00.175387 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:00.175366    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c85847677-kgp5f"
Jul 29 21:00:00.288369 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:00.288334    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z"]
Jul 29 21:00:00.291479 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:00:00.291442    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0444bdab_349d_4148_b08c_f588110c750d.slice/crio-cde4166de8dfbbd9e2e976f89e4f11f497f4413c1bcd151a18d8ff16c9a7480b WatchSource:0}: Error finding container cde4166de8dfbbd9e2e976f89e4f11f497f4413c1bcd151a18d8ff16c9a7480b: Status 404 returned error can't find the container with id cde4166de8dfbbd9e2e976f89e4f11f497f4413c1bcd151a18d8ff16c9a7480b
Jul 29 21:00:01.228624 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:01.228591    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z" event={"ID":"0444bdab-349d-4148-b08c-f588110c750d","Type":"ContainerStarted","Data":"cde4166de8dfbbd9e2e976f89e4f11f497f4413c1bcd151a18d8ff16c9a7480b"}
Jul 29 21:00:03.239400 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:03.239365    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z" event={"ID":"0444bdab-349d-4148-b08c-f588110c750d","Type":"ContainerStarted","Data":"dcb378cb597ee8b82c31aa6e6ff1082dbfc71b4dfb0c866668ccdd6b7d829e05"}
Jul 29 21:00:03.254958 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:03.254865    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z" podStartSLOduration=0.882487022 podStartE2EDuration="3.254851721s" totalImagesPullingTime="2.372364699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:00:03.252876645 +0000 UTC m=+818.849168328" watchObservedRunningTime="2026-07-29 21:00:03.254851721 +0000 UTC m=+818.851143404"
Jul 29 21:00:04.244767 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:04.244728    2584 generic.go:356] "Generic (PLEG): container finished" podID="0444bdab-349d-4148-b08c-f588110c750d" containerID="dcb378cb597ee8b82c31aa6e6ff1082dbfc71b4dfb0c866668ccdd6b7d829e05" exitCode=0
Jul 29 21:00:04.245126 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:04.244779    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z" event={"ID":"0444bdab-349d-4148-b08c-f588110c750d","Type":"ContainerDied","Data":"dcb378cb597ee8b82c31aa6e6ff1082dbfc71b4dfb0c866668ccdd6b7d829e05"}
Jul 29 21:00:04.844089 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:04.844054    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c85847677-kgp5f"]
Jul 29 21:00:04.844303 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:04.844255    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c85847677-kgp5f" podUID="8b49a4bd-6383-43b5-bd36-147408502e8d" containerName="manager" containerID="cri-o://b93912cac5502fd74d01fe09578b29734bc9c0128fe00be3d11471ca25d41625" gracePeriod=30
Jul 29 21:00:05.092731 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.092703    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c85847677-kgp5f"
Jul 29 21:00:05.163459 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.163368    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b49a4bd-6383-43b5-bd36-147408502e8d-webhook-cert\") pod \"8b49a4bd-6383-43b5-bd36-147408502e8d\" (UID: \"8b49a4bd-6383-43b5-bd36-147408502e8d\") "
Jul 29 21:00:05.163459 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.163421    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2ls4\" (UniqueName: \"kubernetes.io/projected/8b49a4bd-6383-43b5-bd36-147408502e8d-kube-api-access-w2ls4\") pod \"8b49a4bd-6383-43b5-bd36-147408502e8d\" (UID: \"8b49a4bd-6383-43b5-bd36-147408502e8d\") "
Jul 29 21:00:05.165519 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.165491    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b49a4bd-6383-43b5-bd36-147408502e8d-kube-api-access-w2ls4" (OuterVolumeSpecName: "kube-api-access-w2ls4") pod "8b49a4bd-6383-43b5-bd36-147408502e8d" (UID: "8b49a4bd-6383-43b5-bd36-147408502e8d"). InnerVolumeSpecName "kube-api-access-w2ls4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:00:05.165624 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.165520    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b49a4bd-6383-43b5-bd36-147408502e8d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8b49a4bd-6383-43b5-bd36-147408502e8d" (UID: "8b49a4bd-6383-43b5-bd36-147408502e8d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:00:05.250379 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.250347    2584 generic.go:356] "Generic (PLEG): container finished" podID="8b49a4bd-6383-43b5-bd36-147408502e8d" containerID="b93912cac5502fd74d01fe09578b29734bc9c0128fe00be3d11471ca25d41625" exitCode=0
Jul 29 21:00:05.250770 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.250388    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c85847677-kgp5f"
Jul 29 21:00:05.250770 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.250424    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c85847677-kgp5f" event={"ID":"8b49a4bd-6383-43b5-bd36-147408502e8d","Type":"ContainerDied","Data":"b93912cac5502fd74d01fe09578b29734bc9c0128fe00be3d11471ca25d41625"}
Jul 29 21:00:05.250770 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.250467    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c85847677-kgp5f" event={"ID":"8b49a4bd-6383-43b5-bd36-147408502e8d","Type":"ContainerDied","Data":"9c6c8048fe59233c61ee9226ec9fbf6973df58521417019b96a2f89292b0d215"}
Jul 29 21:00:05.250770 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.250490    2584 scope.go:117] "RemoveContainer" containerID="b93912cac5502fd74d01fe09578b29734bc9c0128fe00be3d11471ca25d41625"
Jul 29 21:00:05.261182 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.260682    2584 scope.go:117] "RemoveContainer" containerID="b93912cac5502fd74d01fe09578b29734bc9c0128fe00be3d11471ca25d41625"
Jul 29 21:00:05.261182 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:00:05.261098    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93912cac5502fd74d01fe09578b29734bc9c0128fe00be3d11471ca25d41625\": container with ID starting with b93912cac5502fd74d01fe09578b29734bc9c0128fe00be3d11471ca25d41625 not found: ID does not exist" containerID="b93912cac5502fd74d01fe09578b29734bc9c0128fe00be3d11471ca25d41625"
Jul 29 21:00:05.261182 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.261137    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93912cac5502fd74d01fe09578b29734bc9c0128fe00be3d11471ca25d41625"} err="failed to get container status \"b93912cac5502fd74d01fe09578b29734bc9c0128fe00be3d11471ca25d41625\": rpc error: code = NotFound desc = could not find container \"b93912cac5502fd74d01fe09578b29734bc9c0128fe00be3d11471ca25d41625\": container with ID starting with b93912cac5502fd74d01fe09578b29734bc9c0128fe00be3d11471ca25d41625 not found: ID does not exist"
Jul 29 21:00:05.264649 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.264615    2584 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b49a4bd-6383-43b5-bd36-147408502e8d-webhook-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:00:05.264649 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.264642    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2ls4\" (UniqueName: \"kubernetes.io/projected/8b49a4bd-6383-43b5-bd36-147408502e8d-kube-api-access-w2ls4\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:00:05.277368 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.277339    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c85847677-kgp5f"]
Jul 29 21:00:05.282354 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.282329    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c85847677-kgp5f"]
Jul 29 21:00:05.375426 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:05.375400    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z"
Jul 29 21:00:06.051237 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.051200    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c85847677-c7fqc"]
Jul 29 21:00:06.051536 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.051523    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0444bdab-349d-4148-b08c-f588110c750d" containerName="cleanup"
Jul 29 21:00:06.051579 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.051537    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0444bdab-349d-4148-b08c-f588110c750d" containerName="cleanup"
Jul 29 21:00:06.051579 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.051553    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b49a4bd-6383-43b5-bd36-147408502e8d" containerName="manager"
Jul 29 21:00:06.051579 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.051559    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b49a4bd-6383-43b5-bd36-147408502e8d" containerName="manager"
Jul 29 21:00:06.051674 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.051625    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b49a4bd-6383-43b5-bd36-147408502e8d" containerName="manager"
Jul 29 21:00:06.051674 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.051633    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0444bdab-349d-4148-b08c-f588110c750d" containerName="cleanup"
Jul 29 21:00:06.054219 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.054200    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c85847677-c7fqc"
Jul 29 21:00:06.056608 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.056583    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 21:00:06.056708 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.056586    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pvk7c\""
Jul 29 21:00:06.056708 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.056590    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 21:00:06.064940 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.064919    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c85847677-c7fqc"]
Jul 29 21:00:06.171296 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.171202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f3f9a7e-2441-476a-8f8d-1b72f46a2384-webhook-cert\") pod \"maas-controller-5c85847677-c7fqc\" (UID: \"6f3f9a7e-2441-476a-8f8d-1b72f46a2384\") " pod="opendatahub/maas-controller-5c85847677-c7fqc"
Jul 29 21:00:06.171455 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.171318    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xcp\" (UniqueName: \"kubernetes.io/projected/6f3f9a7e-2441-476a-8f8d-1b72f46a2384-kube-api-access-z8xcp\") pod \"maas-controller-5c85847677-c7fqc\" (UID: \"6f3f9a7e-2441-476a-8f8d-1b72f46a2384\") " pod="opendatahub/maas-controller-5c85847677-c7fqc"
Jul 29 21:00:06.255933 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.255912    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z"
Jul 29 21:00:06.255933 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.255925    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z" event={"ID":"0444bdab-349d-4148-b08c-f588110c750d","Type":"ContainerDied","Data":"cde4166de8dfbbd9e2e976f89e4f11f497f4413c1bcd151a18d8ff16c9a7480b"}
Jul 29 21:00:06.256451 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.255966    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cde4166de8dfbbd9e2e976f89e4f11f497f4413c1bcd151a18d8ff16c9a7480b"
Jul 29 21:00:06.272502 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.272480    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xcp\" (UniqueName: \"kubernetes.io/projected/6f3f9a7e-2441-476a-8f8d-1b72f46a2384-kube-api-access-z8xcp\") pod \"maas-controller-5c85847677-c7fqc\" (UID: \"6f3f9a7e-2441-476a-8f8d-1b72f46a2384\") " pod="opendatahub/maas-controller-5c85847677-c7fqc"
Jul 29 21:00:06.272611 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.272532    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f3f9a7e-2441-476a-8f8d-1b72f46a2384-webhook-cert\") pod \"maas-controller-5c85847677-c7fqc\" (UID: \"6f3f9a7e-2441-476a-8f8d-1b72f46a2384\") " pod="opendatahub/maas-controller-5c85847677-c7fqc"
Jul 29 21:00:06.274930 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.274907    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f3f9a7e-2441-476a-8f8d-1b72f46a2384-webhook-cert\") pod \"maas-controller-5c85847677-c7fqc\" (UID: \"6f3f9a7e-2441-476a-8f8d-1b72f46a2384\") " pod="opendatahub/maas-controller-5c85847677-c7fqc"
Jul 29 21:00:06.280050 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.280017    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xcp\" (UniqueName: \"kubernetes.io/projected/6f3f9a7e-2441-476a-8f8d-1b72f46a2384-kube-api-access-z8xcp\") pod \"maas-controller-5c85847677-c7fqc\" (UID: \"6f3f9a7e-2441-476a-8f8d-1b72f46a2384\") " pod="opendatahub/maas-controller-5c85847677-c7fqc"
Jul 29 21:00:06.364137 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.364103    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c85847677-c7fqc"
Jul 29 21:00:06.488256 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.488229    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c85847677-c7fqc"]
Jul 29 21:00:06.490220 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:00:06.490195    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3f9a7e_2441_476a_8f8d_1b72f46a2384.slice/crio-260b11a37228016a38e1e9feab4c4cac6e16c86a133d9123bed79f761262b667 WatchSource:0}: Error finding container 260b11a37228016a38e1e9feab4c4cac6e16c86a133d9123bed79f761262b667: Status 404 returned error can't find the container with id 260b11a37228016a38e1e9feab4c4cac6e16c86a133d9123bed79f761262b667
Jul 29 21:00:06.865560 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:06.865526    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b49a4bd-6383-43b5-bd36-147408502e8d" path="/var/lib/kubelet/pods/8b49a4bd-6383-43b5-bd36-147408502e8d/volumes"
Jul 29 21:00:07.262200 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:07.262120    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c85847677-c7fqc" event={"ID":"6f3f9a7e-2441-476a-8f8d-1b72f46a2384","Type":"ContainerStarted","Data":"260b11a37228016a38e1e9feab4c4cac6e16c86a133d9123bed79f761262b667"}
Jul 29 21:00:08.267293 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:08.267255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c85847677-c7fqc" event={"ID":"6f3f9a7e-2441-476a-8f8d-1b72f46a2384","Type":"ContainerStarted","Data":"cc0f41d5c4321a01d8227613fb2c62f355a02820606dbdd9f32ca49a5c1df454"}
Jul 29 21:00:08.267681 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:08.267318    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c85847677-c7fqc"
Jul 29 21:00:08.283129 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:08.283083    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c85847677-c7fqc" podStartSLOduration=1.316473216 podStartE2EDuration="2.283068006s" totalImagesPullingTime="966.59479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:00:08.281859874 +0000 UTC m=+823.878151557" watchObservedRunningTime="2026-07-29 21:00:08.283068006 +0000 UTC m=+823.879359689"
Jul 29 21:00:19.273230 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:00:19.273199    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c85847677-c7fqc"
Jul 29 21:11:14.936416 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:14.936387    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"]
Jul 29 21:11:14.938532 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:14.938515    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:14.940337 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:14.940304    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e53236bf-kserve-self-signed-certs\""
Jul 29 21:11:14.947740 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:14.947719    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"]
Jul 29 21:11:14.973486 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:14.973463    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-model-cache\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:14.973627 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:14.973499    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-home\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:14.973627 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:14.973517    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-tmp-dir\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:14.973627 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:14.973574    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-dshm\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:14.973627 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:14.973620    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6brq\" (UniqueName: \"kubernetes.io/projected/2ba91172-ba8b-42b1-b25c-ec618828edaf-kube-api-access-l6brq\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:14.973761 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:14.973641    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba91172-ba8b-42b1-b25c-ec618828edaf-tls-certs\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.075033 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.075000    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6brq\" (UniqueName: \"kubernetes.io/projected/2ba91172-ba8b-42b1-b25c-ec618828edaf-kube-api-access-l6brq\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.075033 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.075035    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba91172-ba8b-42b1-b25c-ec618828edaf-tls-certs\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.075304 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.075064    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-model-cache\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.075304 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.075088    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-home\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.075304 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.075106    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-tmp-dir\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.075304 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.075134    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-dshm\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.075554 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.075533    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-model-cache\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.075554 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.075546    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-home\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.075670 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.075619    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-tmp-dir\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.077531 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.077511    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-dshm\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.077696 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.077667    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba91172-ba8b-42b1-b25c-ec618828edaf-tls-certs\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.082207 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.082188    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6brq\" (UniqueName: \"kubernetes.io/projected/2ba91172-ba8b-42b1-b25c-ec618828edaf-kube-api-access-l6brq\") pod \"e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") " pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.248804 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.248725    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:15.374172 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:15.374087    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"]
Jul 29 21:11:15.376547 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:11:15.376522    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ba91172_ba8b_42b1_b25c_ec618828edaf.slice/crio-ea603e04f9fe9621dacb8c90bb38012c5f8141e393040b838ee59ac28a655dc9 WatchSource:0}: Error finding container ea603e04f9fe9621dacb8c90bb38012c5f8141e393040b838ee59ac28a655dc9: Status 404 returned error can't find the container with id ea603e04f9fe9621dacb8c90bb38012c5f8141e393040b838ee59ac28a655dc9
Jul 29 21:11:16.009365 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:16.009320    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8" event={"ID":"2ba91172-ba8b-42b1-b25c-ec618828edaf","Type":"ContainerStarted","Data":"9095070beb6a6c5d5107d7b42637e11f5467c59064b8e47430c1fe8cb8b7fac9"}
Jul 29 21:11:16.009759 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:16.009369    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8" event={"ID":"2ba91172-ba8b-42b1-b25c-ec618828edaf","Type":"ContainerStarted","Data":"ea603e04f9fe9621dacb8c90bb38012c5f8141e393040b838ee59ac28a655dc9"}
Jul 29 21:11:16.024341 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:16.024291    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8" podStartSLOduration=2.024270904 podStartE2EDuration="2.024270904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:11:16.022979271 +0000 UTC m=+1491.619270958" watchObservedRunningTime="2026-07-29 21:11:16.024270904 +0000 UTC m=+1491.620562588"
Jul 29 21:11:25.249395 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:25.249362    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:25.249395 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:25.249404    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:25.262996 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:25.262970    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:26.061915 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:26.061888    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:41.261094 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:41.261063    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"]
Jul 29 21:11:56.280920 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.280839    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8" podUID="2ba91172-ba8b-42b1-b25c-ec618828edaf" containerName="main" containerID="cri-o://9095070beb6a6c5d5107d7b42637e11f5467c59064b8e47430c1fe8cb8b7fac9" gracePeriod=45
Jul 29 21:11:56.523199 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.523173    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:56.623137 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.623105    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-home\") pod \"2ba91172-ba8b-42b1-b25c-ec618828edaf\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") "
Jul 29 21:11:56.623330 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.623183    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-dshm\") pod \"2ba91172-ba8b-42b1-b25c-ec618828edaf\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") "
Jul 29 21:11:56.623330 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.623211    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-model-cache\") pod \"2ba91172-ba8b-42b1-b25c-ec618828edaf\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") "
Jul 29 21:11:56.623330 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.623234    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-tmp-dir\") pod \"2ba91172-ba8b-42b1-b25c-ec618828edaf\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") "
Jul 29 21:11:56.623330 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.623262    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba91172-ba8b-42b1-b25c-ec618828edaf-tls-certs\") pod \"2ba91172-ba8b-42b1-b25c-ec618828edaf\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") "
Jul 29 21:11:56.623330 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.623307    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6brq\" (UniqueName: \"kubernetes.io/projected/2ba91172-ba8b-42b1-b25c-ec618828edaf-kube-api-access-l6brq\") pod \"2ba91172-ba8b-42b1-b25c-ec618828edaf\" (UID: \"2ba91172-ba8b-42b1-b25c-ec618828edaf\") "
Jul 29 21:11:56.623556 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.623384    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-home" (OuterVolumeSpecName: "home") pod "2ba91172-ba8b-42b1-b25c-ec618828edaf" (UID: "2ba91172-ba8b-42b1-b25c-ec618828edaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:11:56.623556 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.623456    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ba91172-ba8b-42b1-b25c-ec618828edaf" (UID: "2ba91172-ba8b-42b1-b25c-ec618828edaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:11:56.623556 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.623472    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ba91172-ba8b-42b1-b25c-ec618828edaf" (UID: "2ba91172-ba8b-42b1-b25c-ec618828edaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:11:56.623650 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.623581    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:11:56.623650 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.623600    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:11:56.623650 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.623615    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:11:56.625426 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.625396    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-dshm" (OuterVolumeSpecName: "dshm") pod "2ba91172-ba8b-42b1-b25c-ec618828edaf" (UID: "2ba91172-ba8b-42b1-b25c-ec618828edaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:11:56.625526 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.625464    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba91172-ba8b-42b1-b25c-ec618828edaf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ba91172-ba8b-42b1-b25c-ec618828edaf" (UID: "2ba91172-ba8b-42b1-b25c-ec618828edaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:11:56.625526 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.625488    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba91172-ba8b-42b1-b25c-ec618828edaf-kube-api-access-l6brq" (OuterVolumeSpecName: "kube-api-access-l6brq") pod "2ba91172-ba8b-42b1-b25c-ec618828edaf" (UID: "2ba91172-ba8b-42b1-b25c-ec618828edaf"). InnerVolumeSpecName "kube-api-access-l6brq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:11:56.724727 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.724692    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba91172-ba8b-42b1-b25c-ec618828edaf-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:11:56.724727 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.724721    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba91172-ba8b-42b1-b25c-ec618828edaf-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:11:56.724727 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:56.724732    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6brq\" (UniqueName: \"kubernetes.io/projected/2ba91172-ba8b-42b1-b25c-ec618828edaf-kube-api-access-l6brq\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:11:57.182483 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:57.182450    2584 generic.go:356] "Generic (PLEG): container finished" podID="2ba91172-ba8b-42b1-b25c-ec618828edaf" containerID="9095070beb6a6c5d5107d7b42637e11f5467c59064b8e47430c1fe8cb8b7fac9" exitCode=0
Jul 29 21:11:57.182483 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:57.182487    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8" event={"ID":"2ba91172-ba8b-42b1-b25c-ec618828edaf","Type":"ContainerDied","Data":"9095070beb6a6c5d5107d7b42637e11f5467c59064b8e47430c1fe8cb8b7fac9"}
Jul 29 21:11:57.182702 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:57.182491    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"
Jul 29 21:11:57.182702 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:57.182513    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8" event={"ID":"2ba91172-ba8b-42b1-b25c-ec618828edaf","Type":"ContainerDied","Data":"ea603e04f9fe9621dacb8c90bb38012c5f8141e393040b838ee59ac28a655dc9"}
Jul 29 21:11:57.182702 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:57.182531    2584 scope.go:117] "RemoveContainer" containerID="9095070beb6a6c5d5107d7b42637e11f5467c59064b8e47430c1fe8cb8b7fac9"
Jul 29 21:11:57.190494 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:57.190477    2584 scope.go:117] "RemoveContainer" containerID="9095070beb6a6c5d5107d7b42637e11f5467c59064b8e47430c1fe8cb8b7fac9"
Jul 29 21:11:57.190767 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:11:57.190745    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9095070beb6a6c5d5107d7b42637e11f5467c59064b8e47430c1fe8cb8b7fac9\": container with ID starting with 9095070beb6a6c5d5107d7b42637e11f5467c59064b8e47430c1fe8cb8b7fac9 not found: ID does not exist" containerID="9095070beb6a6c5d5107d7b42637e11f5467c59064b8e47430c1fe8cb8b7fac9"
Jul 29 21:11:57.190823 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:57.190776    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9095070beb6a6c5d5107d7b42637e11f5467c59064b8e47430c1fe8cb8b7fac9"} err="failed to get container status \"9095070beb6a6c5d5107d7b42637e11f5467c59064b8e47430c1fe8cb8b7fac9\": rpc error: code = NotFound desc = could not find container \"9095070beb6a6c5d5107d7b42637e11f5467c59064b8e47430c1fe8cb8b7fac9\": container with ID starting with 9095070beb6a6c5d5107d7b42637e11f5467c59064b8e47430c1fe8cb8b7fac9 not found: ID does not exist"
Jul 29 21:11:57.196738 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:57.196718    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"]
Jul 29 21:11:57.198584 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:57.198562    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e53236bf-kserve-58548cf987-d26s8"]
Jul 29 21:11:58.864347 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:11:58.864313    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ba91172-ba8b-42b1-b25c-ec618828edaf" path="/var/lib/kubelet/pods/2ba91172-ba8b-42b1-b25c-ec618828edaf/volumes"
Jul 29 21:15:00.140272 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:00.140196    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm"]
Jul 29 21:15:00.140719 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:00.140512    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba91172-ba8b-42b1-b25c-ec618828edaf" containerName="main"
Jul 29 21:15:00.140719 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:00.140522    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba91172-ba8b-42b1-b25c-ec618828edaf" containerName="main"
Jul 29 21:15:00.140719 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:00.140585    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba91172-ba8b-42b1-b25c-ec618828edaf" containerName="main"
Jul 29 21:15:00.143431 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:00.143416    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm"
Jul 29 21:15:00.145400 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:00.145381    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\""
Jul 29 21:15:00.152707 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:00.152690    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm"
Jul 29 21:15:00.158295 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:00.158273    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm"]
Jul 29 21:15:00.278266 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:00.278204    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm"]
Jul 29 21:15:00.280562 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:15:00.280534    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac90060_c993_441c_b079_b124691cbc79.slice/crio-e02842fe5221ed084f6267f85f8fc3218d8ae1c890413cd68aafcf7657e57e96 WatchSource:0}: Error finding container e02842fe5221ed084f6267f85f8fc3218d8ae1c890413cd68aafcf7657e57e96: Status 404 returned error can't find the container with id e02842fe5221ed084f6267f85f8fc3218d8ae1c890413cd68aafcf7657e57e96
Jul 29 21:15:00.945902 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:00.945867    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm" event={"ID":"8ac90060-c993-441c-b079-b124691cbc79","Type":"ContainerStarted","Data":"eb6bd3b9713a4f31eb1e85b5bc126f199afe6e3e93f08a0c5f90e8b7d28fb94e"}
Jul 29 21:15:00.946077 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:00.945931    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm" event={"ID":"8ac90060-c993-441c-b079-b124691cbc79","Type":"ContainerStarted","Data":"e02842fe5221ed084f6267f85f8fc3218d8ae1c890413cd68aafcf7657e57e96"}
Jul 29 21:15:00.965006 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:00.962862    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm" podStartSLOduration=0.96283924 podStartE2EDuration="962.83924ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:15:00.959188288 +0000 UTC m=+1716.555479996" watchObservedRunningTime="2026-07-29 21:15:00.96283924 +0000 UTC m=+1716.559130926"
Jul 29 21:15:01.951089 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:01.951058    2584 generic.go:356] "Generic (PLEG): container finished" podID="8ac90060-c993-441c-b079-b124691cbc79" containerID="eb6bd3b9713a4f31eb1e85b5bc126f199afe6e3e93f08a0c5f90e8b7d28fb94e" exitCode=0
Jul 29 21:15:01.951507 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:01.951132    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm" event={"ID":"8ac90060-c993-441c-b079-b124691cbc79","Type":"ContainerDied","Data":"eb6bd3b9713a4f31eb1e85b5bc126f199afe6e3e93f08a0c5f90e8b7d28fb94e"}
Jul 29 21:15:03.082387 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:03.082364    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm"
Jul 29 21:15:03.960910 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:03.960880    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm"
Jul 29 21:15:03.960910 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:03.960889    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm" event={"ID":"8ac90060-c993-441c-b079-b124691cbc79","Type":"ContainerDied","Data":"e02842fe5221ed084f6267f85f8fc3218d8ae1c890413cd68aafcf7657e57e96"}
Jul 29 21:15:03.961112 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:15:03.960941    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e02842fe5221ed084f6267f85f8fc3218d8ae1c890413cd68aafcf7657e57e96"
Jul 29 21:16:23.264514 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.264477    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"]
Jul 29 21:16:23.264902 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.264812    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac90060-c993-441c-b079-b124691cbc79" containerName="cleanup"
Jul 29 21:16:23.264902 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.264824    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac90060-c993-441c-b079-b124691cbc79" containerName="cleanup"
Jul 29 21:16:23.264902 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.264873    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac90060-c993-441c-b079-b124691cbc79" containerName="cleanup"
Jul 29 21:16:23.267717 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.267696    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.269594 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.269574    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-23c87980-openshift-default-dockercfg-dfs7h\""
Jul 29 21:16:23.281118 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.281089    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"]
Jul 29 21:16:23.438623 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.438592    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-credential-socket\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.438794 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.438632    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-workload-socket\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.438794 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.438656    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81739c1-7295-4b75-9179-94dd48131d6b-istiod-ca-cert\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.438794 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.438693    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9kp\" (UniqueName: \"kubernetes.io/projected/f81739c1-7295-4b75-9179-94dd48131d6b-kube-api-access-xq9kp\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.438794 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.438764    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-istio-envoy\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.438794 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.438785    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-istio-data\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.439010 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.438800    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81739c1-7295-4b75-9179-94dd48131d6b-istio-podinfo\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.439010 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.438857    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f81739c1-7295-4b75-9179-94dd48131d6b-istio-token\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.439010 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.438888    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-workload-certs\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.539880 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.539806    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-credential-socket\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.539880 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.539848    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-workload-socket\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.539880 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.539871    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81739c1-7295-4b75-9179-94dd48131d6b-istiod-ca-cert\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.540167 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.539887    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9kp\" (UniqueName: \"kubernetes.io/projected/f81739c1-7295-4b75-9179-94dd48131d6b-kube-api-access-xq9kp\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.540167 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.539907    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-istio-envoy\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.540167 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.539927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-istio-data\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.540167 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.540039    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81739c1-7295-4b75-9179-94dd48131d6b-istio-podinfo\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.540167 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.540095    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f81739c1-7295-4b75-9179-94dd48131d6b-istio-token\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.540167 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.540122    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-workload-certs\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.540485 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.540327    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-workload-socket\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.540485 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.540409    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-istio-data\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.540485 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.540430    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-credential-socket\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.540618 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.540500    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-workload-certs\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.540699 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.540681    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81739c1-7295-4b75-9179-94dd48131d6b-istiod-ca-cert\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.542307 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.542285    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-istio-envoy\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.542536 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.542517    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81739c1-7295-4b75-9179-94dd48131d6b-istio-podinfo\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.546732 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.546709    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f81739c1-7295-4b75-9179-94dd48131d6b-istio-token\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.546943 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.546926    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9kp\" (UniqueName: \"kubernetes.io/projected/f81739c1-7295-4b75-9179-94dd48131d6b-kube-api-access-xq9kp\") pod \"e2e-ait-23c87980-openshift-default-7765cd684-hbr86\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") " pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.577868 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.577847    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:23.707417 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.707392    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"]
Jul 29 21:16:23.709868 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:16:23.709836    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81739c1_7295_4b75_9179_94dd48131d6b.slice/crio-e33d87d77428d0daa256748f5ec9d1fbb30320fda9e10460fac7cc767cbe5952 WatchSource:0}: Error finding container e33d87d77428d0daa256748f5ec9d1fbb30320fda9e10460fac7cc767cbe5952: Status 404 returned error can't find the container with id e33d87d77428d0daa256748f5ec9d1fbb30320fda9e10460fac7cc767cbe5952
Jul 29 21:16:23.711983 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.711955    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:16:23.712055 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.712013    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:16:23.712055 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:23.712036    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:16:24.293590 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:24.293556    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" event={"ID":"f81739c1-7295-4b75-9179-94dd48131d6b","Type":"ContainerStarted","Data":"ed468eaa11c3574e7f73f0ca55c08d8d6307b9437f37dafdecfc559339f71409"}
Jul 29 21:16:24.294019 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:24.293614    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" event={"ID":"f81739c1-7295-4b75-9179-94dd48131d6b","Type":"ContainerStarted","Data":"e33d87d77428d0daa256748f5ec9d1fbb30320fda9e10460fac7cc767cbe5952"}
Jul 29 21:16:24.312929 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:24.312885    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" podStartSLOduration=1.312871328 podStartE2EDuration="1.312871328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:24.310572484 +0000 UTC m=+1799.906864179" watchObservedRunningTime="2026-07-29 21:16:24.312871328 +0000 UTC m=+1799.909163012"
Jul 29 21:16:24.578579 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:24.578497    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:24.583245 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:24.583220    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:25.298593 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:25.298560    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:25.299651 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:25.299631    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:40.995285 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:40.994871    2584 patch_prober.go:28] interesting pod/e2e-ait-23c87980-openshift-default-7765cd684-hbr86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": read tcp 10.133.0.2:42370->10.133.0.35:15021: read: connection reset by peer" start-of-body=
Jul 29 21:16:40.995285 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:40.994951    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": read tcp 10.133.0.2:42370->10.133.0.35:15021: read: connection reset by peer"
Jul 29 21:16:41.369331 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:41.369300    2584 generic.go:356] "Generic (PLEG): container finished" podID="f81739c1-7295-4b75-9179-94dd48131d6b" containerID="ed468eaa11c3574e7f73f0ca55c08d8d6307b9437f37dafdecfc559339f71409" exitCode=0
Jul 29 21:16:41.369510 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:41.369376    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" event={"ID":"f81739c1-7295-4b75-9179-94dd48131d6b","Type":"ContainerDied","Data":"ed468eaa11c3574e7f73f0ca55c08d8d6307b9437f37dafdecfc559339f71409"}
Jul 29 21:16:41.369788 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:41.369772    2584 scope.go:117] "RemoveContainer" containerID="ed468eaa11c3574e7f73f0ca55c08d8d6307b9437f37dafdecfc559339f71409"
Jul 29 21:16:41.370761 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:41.370738    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:16:41.370820 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:41.370804    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:16:41.370855 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:41.370834    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:16:42.375437 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:42.375407    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" event={"ID":"f81739c1-7295-4b75-9179-94dd48131d6b","Type":"ContainerStarted","Data":"f7b1aac8811324e7d2a153928398a6c74cbb3fcdd9359badc511226bb9e50144"}
Jul 29 21:16:42.578085 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:42.578043    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:42.578329 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:42.578303    2584 patch_prober.go:28] interesting pod/e2e-ait-23c87980-openshift-default-7765cd684-hbr86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 29 21:16:42.578394 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:42.578371    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 29 21:16:43.578396 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:43.578359    2584 patch_prober.go:28] interesting pod/e2e-ait-23c87980-openshift-default-7765cd684-hbr86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 29 21:16:43.578849 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:43.578432    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 29 21:16:44.578786 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:44.578744    2584 patch_prober.go:28] interesting pod/e2e-ait-23c87980-openshift-default-7765cd684-hbr86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 29 21:16:44.579193 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:44.578808    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 29 21:16:45.578931 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:45.578892    2584 patch_prober.go:28] interesting pod/e2e-ait-23c87980-openshift-default-7765cd684-hbr86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 29 21:16:45.579421 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:45.578950    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 29 21:16:46.582417 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:46.582385    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:46.582784 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:46.582662    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:46.583331 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:46.583308    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:16:51.239429 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:51.239391    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj"]
Jul 29 21:16:51.241648 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:51.241630    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj"
Jul 29 21:16:51.243958 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:51.243933    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:16:51.243958 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:51.243945    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\""
Jul 29 21:16:51.250046 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:51.250022    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj"]
Jul 29 21:16:51.269807 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:51.269783    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3113b226-fa87-4649-9112-b3e9e2db893c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23c87980-c88zj\" (UID: \"3113b226-fa87-4649-9112-b3e9e2db893c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj"
Jul 29 21:16:51.370346 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:51.370318    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3113b226-fa87-4649-9112-b3e9e2db893c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23c87980-c88zj\" (UID: \"3113b226-fa87-4649-9112-b3e9e2db893c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj"
Jul 29 21:16:51.370916 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:51.370898    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3113b226-fa87-4649-9112-b3e9e2db893c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23c87980-c88zj\" (UID: \"3113b226-fa87-4649-9112-b3e9e2db893c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj"
Jul 29 21:16:51.553631 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:51.553558    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj"
Jul 29 21:16:51.674592 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:51.674565    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj"]
Jul 29 21:16:51.677199 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:16:51.677175    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3113b226_fa87_4649_9112_b3e9e2db893c.slice/crio-86211636ce3aa556a5bf009056fd12414aa73715d617c60959ee06574e788ce2 WatchSource:0}: Error finding container 86211636ce3aa556a5bf009056fd12414aa73715d617c60959ee06574e788ce2: Status 404 returned error can't find the container with id 86211636ce3aa556a5bf009056fd12414aa73715d617c60959ee06574e788ce2
Jul 29 21:16:52.423095 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:52.423054    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj" event={"ID":"3113b226-fa87-4649-9112-b3e9e2db893c","Type":"ContainerStarted","Data":"5e0fc1fd745fec4143604c2d0ae32bf9750096927d281765170a5cb883dc4251"}
Jul 29 21:16:52.423486 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:52.423121    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj" event={"ID":"3113b226-fa87-4649-9112-b3e9e2db893c","Type":"ContainerStarted","Data":"86211636ce3aa556a5bf009056fd12414aa73715d617c60959ee06574e788ce2"}
Jul 29 21:16:52.437995 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:52.437950    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj" podStartSLOduration=1.437935047 podStartE2EDuration="1.437935047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:52.436604083 +0000 UTC m=+1828.032895767" watchObservedRunningTime="2026-07-29 21:16:52.437935047 +0000 UTC m=+1828.034226752"
Jul 29 21:16:53.428894 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:53.428857    2584 generic.go:356] "Generic (PLEG): container finished" podID="3113b226-fa87-4649-9112-b3e9e2db893c" containerID="5e0fc1fd745fec4143604c2d0ae32bf9750096927d281765170a5cb883dc4251" exitCode=0
Jul 29 21:16:53.429350 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:53.428908    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj" event={"ID":"3113b226-fa87-4649-9112-b3e9e2db893c","Type":"ContainerDied","Data":"5e0fc1fd745fec4143604c2d0ae32bf9750096927d281765170a5cb883dc4251"}
Jul 29 21:16:54.570086 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:54.570061    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj"
Jul 29 21:16:54.595844 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:54.595818    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3113b226-fa87-4649-9112-b3e9e2db893c-maas-api-service-ca\") pod \"3113b226-fa87-4649-9112-b3e9e2db893c\" (UID: \"3113b226-fa87-4649-9112-b3e9e2db893c\") "
Jul 29 21:16:54.596260 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:54.596235    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3113b226-fa87-4649-9112-b3e9e2db893c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3113b226-fa87-4649-9112-b3e9e2db893c" (UID: "3113b226-fa87-4649-9112-b3e9e2db893c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:16:54.696797 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:54.696726    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3113b226-fa87-4649-9112-b3e9e2db893c-maas-api-service-ca\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:16:55.439422 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:55.439383    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj" event={"ID":"3113b226-fa87-4649-9112-b3e9e2db893c","Type":"ContainerDied","Data":"86211636ce3aa556a5bf009056fd12414aa73715d617c60959ee06574e788ce2"}
Jul 29 21:16:55.439422 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:55.439413    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj"
Jul 29 21:16:55.439630 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:55.439444    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86211636ce3aa556a5bf009056fd12414aa73715d617c60959ee06574e788ce2"
Jul 29 21:16:57.426535 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:57.426495    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj"]
Jul 29 21:16:57.428766 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:57.428739    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23c87980-c88zj"]
Jul 29 21:16:57.560675 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:57.560644    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"]
Jul 29 21:16:57.560870 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:57.560844    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" containerName="istio-proxy" containerID="cri-o://f7b1aac8811324e7d2a153928398a6c74cbb3fcdd9359badc511226bb9e50144" gracePeriod=30
Jul 29 21:16:58.562491 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.562462    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"]
Jul 29 21:16:58.562874 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.562783    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3113b226-fa87-4649-9112-b3e9e2db893c" containerName="revoke-keys"
Jul 29 21:16:58.562874 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.562794    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3113b226-fa87-4649-9112-b3e9e2db893c" containerName="revoke-keys"
Jul 29 21:16:58.562874 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.562855    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3113b226-fa87-4649-9112-b3e9e2db893c" containerName="revoke-keys"
Jul 29 21:16:58.564852 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.564835    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.567572 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.567552    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-946d3187-gw-openshift-default-dockercfg-hbxvs\""
Jul 29 21:16:58.580802 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.580781    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"]
Jul 29 21:16:58.630773 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.630743    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-workload-socket\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.630773 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.630773    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-token\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.630936 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.630836    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-credential-socket\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.630936 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.630861    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a25a3b1-6478-4651-85fb-6e157654acd9-istiod-ca-cert\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.630936 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.630886    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-data\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.630936 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.630912    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69xb\" (UniqueName: \"kubernetes.io/projected/2a25a3b1-6478-4651-85fb-6e157654acd9-kube-api-access-r69xb\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.630936 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.630930    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-workload-certs\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.631131 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.630956    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-podinfo\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.631131 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.630981    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-envoy\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.731399 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.731358    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-credential-socket\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.731399 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.731401    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a25a3b1-6478-4651-85fb-6e157654acd9-istiod-ca-cert\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.731603 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.731418    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-data\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.731603 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.731543    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69xb\" (UniqueName: \"kubernetes.io/projected/2a25a3b1-6478-4651-85fb-6e157654acd9-kube-api-access-r69xb\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.731603 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.731587    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-workload-certs\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.731752 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.731615    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-podinfo\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.731752 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.731701    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-envoy\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.731851 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.731755    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-data\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.731851 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.731768    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-workload-socket\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.731851 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.731805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-token\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.731997 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.731808    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-credential-socket\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.732370 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.732317    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-workload-certs\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.732510 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.732451    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-workload-socket\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.738510 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.732627    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a25a3b1-6478-4651-85fb-6e157654acd9-istiod-ca-cert\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.738510 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.734883    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-envoy\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.738510 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.735227    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-podinfo\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.741243 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.741224    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-token\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.741328 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.741295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69xb\" (UniqueName: \"kubernetes.io/projected/2a25a3b1-6478-4651-85fb-6e157654acd9-kube-api-access-r69xb\") pod \"e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") " pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.865328 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.865303    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3113b226-fa87-4649-9112-b3e9e2db893c" path="/var/lib/kubelet/pods/3113b226-fa87-4649-9112-b3e9e2db893c/volumes"
Jul 29 21:16:58.874521 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.874498    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:16:58.997869 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:58.997836    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"]
Jul 29 21:16:59.000434 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:16:59.000404    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a25a3b1_6478_4651_85fb_6e157654acd9.slice/crio-04709dee4ac5b02c6c56adca8b90138c0786ce7a057ea1253381becf6bd12489 WatchSource:0}: Error finding container 04709dee4ac5b02c6c56adca8b90138c0786ce7a057ea1253381becf6bd12489: Status 404 returned error can't find the container with id 04709dee4ac5b02c6c56adca8b90138c0786ce7a057ea1253381becf6bd12489
Jul 29 21:16:59.003029 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:59.002999    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:16:59.003096 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:59.003063    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:16:59.003096 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:59.003088    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:16:59.458880 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:59.458846    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8" event={"ID":"2a25a3b1-6478-4651-85fb-6e157654acd9","Type":"ContainerStarted","Data":"8af72b15ff433c9d826c9cc9cc67bab9a79780501e47b01594e61b225b4491c3"}
Jul 29 21:16:59.459038 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:59.458903    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8" event={"ID":"2a25a3b1-6478-4651-85fb-6e157654acd9","Type":"ContainerStarted","Data":"04709dee4ac5b02c6c56adca8b90138c0786ce7a057ea1253381becf6bd12489"}
Jul 29 21:16:59.480293 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:59.480248    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8" podStartSLOduration=1.4802297389999999 podStartE2EDuration="1.480229739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:59.479575903 +0000 UTC m=+1835.075867587" watchObservedRunningTime="2026-07-29 21:16:59.480229739 +0000 UTC m=+1835.076521423"
Jul 29 21:16:59.875343 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:16:59.875314    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:17:00.879776 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:00.879741    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:17:01.467460 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:01.467421    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:17:01.468355 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:01.468335    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:17:01.585235 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:01.585186    2584 patch_prober.go:28] interesting pod/e2e-ait-23c87980-openshift-default-7765cd684-hbr86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 29 21:17:01.585414 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:01.585251    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:17:02.811202 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.811175    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:17:02.869360 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.869330    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-istio-data\") pod \"f81739c1-7295-4b75-9179-94dd48131d6b\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") "
Jul 29 21:17:02.869360 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.869366    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-workload-socket\") pod \"f81739c1-7295-4b75-9179-94dd48131d6b\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") "
Jul 29 21:17:02.869594 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.869417    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-credential-socket\") pod \"f81739c1-7295-4b75-9179-94dd48131d6b\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") "
Jul 29 21:17:02.869594 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.869448    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81739c1-7295-4b75-9179-94dd48131d6b-istio-podinfo\") pod \"f81739c1-7295-4b75-9179-94dd48131d6b\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") "
Jul 29 21:17:02.869594 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.869475    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-workload-certs\") pod \"f81739c1-7295-4b75-9179-94dd48131d6b\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") "
Jul 29 21:17:02.869594 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.869543    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq9kp\" (UniqueName: \"kubernetes.io/projected/f81739c1-7295-4b75-9179-94dd48131d6b-kube-api-access-xq9kp\") pod \"f81739c1-7295-4b75-9179-94dd48131d6b\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") "
Jul 29 21:17:02.869594 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.869569    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f81739c1-7295-4b75-9179-94dd48131d6b-istio-token\") pod \"f81739c1-7295-4b75-9179-94dd48131d6b\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") "
Jul 29 21:17:02.869594 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.869595    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81739c1-7295-4b75-9179-94dd48131d6b-istiod-ca-cert\") pod \"f81739c1-7295-4b75-9179-94dd48131d6b\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") "
Jul 29 21:17:02.869891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.869649    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-istio-envoy\") pod \"f81739c1-7295-4b75-9179-94dd48131d6b\" (UID: \"f81739c1-7295-4b75-9179-94dd48131d6b\") "
Jul 29 21:17:02.869891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.869656    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f81739c1-7295-4b75-9179-94dd48131d6b" (UID: "f81739c1-7295-4b75-9179-94dd48131d6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:17:02.869891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.869738    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f81739c1-7295-4b75-9179-94dd48131d6b" (UID: "f81739c1-7295-4b75-9179-94dd48131d6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:17:02.869891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.869871    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:02.869891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.869889    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:02.870180 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.870127    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f81739c1-7295-4b75-9179-94dd48131d6b" (UID: "f81739c1-7295-4b75-9179-94dd48131d6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:17:02.870257 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.870234    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81739c1-7295-4b75-9179-94dd48131d6b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f81739c1-7295-4b75-9179-94dd48131d6b" (UID: "f81739c1-7295-4b75-9179-94dd48131d6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:17:02.870257 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.870245    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-istio-data" (OuterVolumeSpecName: "istio-data") pod "f81739c1-7295-4b75-9179-94dd48131d6b" (UID: "f81739c1-7295-4b75-9179-94dd48131d6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:17:02.871938 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.871917    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f81739c1-7295-4b75-9179-94dd48131d6b" (UID: "f81739c1-7295-4b75-9179-94dd48131d6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:17:02.872169 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.872127    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f81739c1-7295-4b75-9179-94dd48131d6b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f81739c1-7295-4b75-9179-94dd48131d6b" (UID: "f81739c1-7295-4b75-9179-94dd48131d6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:17:02.872169 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.872136    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81739c1-7295-4b75-9179-94dd48131d6b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f81739c1-7295-4b75-9179-94dd48131d6b" (UID: "f81739c1-7295-4b75-9179-94dd48131d6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:17:02.872257 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.872170    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81739c1-7295-4b75-9179-94dd48131d6b-kube-api-access-xq9kp" (OuterVolumeSpecName: "kube-api-access-xq9kp") pod "f81739c1-7295-4b75-9179-94dd48131d6b" (UID: "f81739c1-7295-4b75-9179-94dd48131d6b"). InnerVolumeSpecName "kube-api-access-xq9kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:17:02.970486 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.970452    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq9kp\" (UniqueName: \"kubernetes.io/projected/f81739c1-7295-4b75-9179-94dd48131d6b-kube-api-access-xq9kp\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:02.970486 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.970480    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f81739c1-7295-4b75-9179-94dd48131d6b-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:02.970486 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.970490    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81739c1-7295-4b75-9179-94dd48131d6b-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:02.970708 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.970498    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:02.970708 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.970506    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:02.970708 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.970518    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81739c1-7295-4b75-9179-94dd48131d6b-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:02.970708 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:02.970528    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81739c1-7295-4b75-9179-94dd48131d6b-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:03.477269 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:03.477234    2584 generic.go:356] "Generic (PLEG): container finished" podID="f81739c1-7295-4b75-9179-94dd48131d6b" containerID="f7b1aac8811324e7d2a153928398a6c74cbb3fcdd9359badc511226bb9e50144" exitCode=0
Jul 29 21:17:03.477436 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:03.477302    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"
Jul 29 21:17:03.477436 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:03.477321    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" event={"ID":"f81739c1-7295-4b75-9179-94dd48131d6b","Type":"ContainerDied","Data":"f7b1aac8811324e7d2a153928398a6c74cbb3fcdd9359badc511226bb9e50144"}
Jul 29 21:17:03.477436 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:03.477380    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86" event={"ID":"f81739c1-7295-4b75-9179-94dd48131d6b","Type":"ContainerDied","Data":"e33d87d77428d0daa256748f5ec9d1fbb30320fda9e10460fac7cc767cbe5952"}
Jul 29 21:17:03.477436 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:03.477403    2584 scope.go:117] "RemoveContainer" containerID="f7b1aac8811324e7d2a153928398a6c74cbb3fcdd9359badc511226bb9e50144"
Jul 29 21:17:03.486903 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:03.486883    2584 scope.go:117] "RemoveContainer" containerID="ed468eaa11c3574e7f73f0ca55c08d8d6307b9437f37dafdecfc559339f71409"
Jul 29 21:17:03.494651 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:03.494633    2584 scope.go:117] "RemoveContainer" containerID="f7b1aac8811324e7d2a153928398a6c74cbb3fcdd9359badc511226bb9e50144"
Jul 29 21:17:03.494890 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:17:03.494873    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b1aac8811324e7d2a153928398a6c74cbb3fcdd9359badc511226bb9e50144\": container with ID starting with f7b1aac8811324e7d2a153928398a6c74cbb3fcdd9359badc511226bb9e50144 not found: ID does not exist" containerID="f7b1aac8811324e7d2a153928398a6c74cbb3fcdd9359badc511226bb9e50144"
Jul 29 21:17:03.494941 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:03.494898    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b1aac8811324e7d2a153928398a6c74cbb3fcdd9359badc511226bb9e50144"} err="failed to get container status \"f7b1aac8811324e7d2a153928398a6c74cbb3fcdd9359badc511226bb9e50144\": rpc error: code = NotFound desc = could not find container \"f7b1aac8811324e7d2a153928398a6c74cbb3fcdd9359badc511226bb9e50144\": container with ID starting with f7b1aac8811324e7d2a153928398a6c74cbb3fcdd9359badc511226bb9e50144 not found: ID does not exist"
Jul 29 21:17:03.494941 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:03.494919    2584 scope.go:117] "RemoveContainer" containerID="ed468eaa11c3574e7f73f0ca55c08d8d6307b9437f37dafdecfc559339f71409"
Jul 29 21:17:03.495123 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:17:03.495106    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed468eaa11c3574e7f73f0ca55c08d8d6307b9437f37dafdecfc559339f71409\": container with ID starting with ed468eaa11c3574e7f73f0ca55c08d8d6307b9437f37dafdecfc559339f71409 not found: ID does not exist" containerID="ed468eaa11c3574e7f73f0ca55c08d8d6307b9437f37dafdecfc559339f71409"
Jul 29 21:17:03.495177 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:03.495129    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed468eaa11c3574e7f73f0ca55c08d8d6307b9437f37dafdecfc559339f71409"} err="failed to get container status \"ed468eaa11c3574e7f73f0ca55c08d8d6307b9437f37dafdecfc559339f71409\": rpc error: code = NotFound desc = could not find container \"ed468eaa11c3574e7f73f0ca55c08d8d6307b9437f37dafdecfc559339f71409\": container with ID starting with ed468eaa11c3574e7f73f0ca55c08d8d6307b9437f37dafdecfc559339f71409 not found: ID does not exist"
Jul 29 21:17:03.503234 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:03.503216    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"]
Jul 29 21:17:03.506253 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:03.506227    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-23c87980-openshift-default-7765cd684-hbr86"]
Jul 29 21:17:04.867219 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:04.867182    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" path="/var/lib/kubelet/pods/f81739c1-7295-4b75-9179-94dd48131d6b/volumes"
Jul 29 21:17:15.537709 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:15.537675    2584 generic.go:356] "Generic (PLEG): container finished" podID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerID="8af72b15ff433c9d826c9cc9cc67bab9a79780501e47b01594e61b225b4491c3" exitCode=0
Jul 29 21:17:15.538098 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:15.537746    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8" event={"ID":"2a25a3b1-6478-4651-85fb-6e157654acd9","Type":"ContainerDied","Data":"8af72b15ff433c9d826c9cc9cc67bab9a79780501e47b01594e61b225b4491c3"}
Jul 29 21:17:15.538167 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:15.538101    2584 scope.go:117] "RemoveContainer" containerID="8af72b15ff433c9d826c9cc9cc67bab9a79780501e47b01594e61b225b4491c3"
Jul 29 21:17:15.539079 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:15.539054    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:17:15.539159 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:15.539114    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:17:15.539215 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:15.539168    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:17:16.468394 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:16.468345    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:17:16.543329 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:16.543296    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8" event={"ID":"2a25a3b1-6478-4651-85fb-6e157654acd9","Type":"ContainerStarted","Data":"02e7877412ebebc6c618637ddfaaf1f459d6f7b782b6532df3c73cf5cf2d15e3"}
Jul 29 21:17:16.875211 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:16.875180    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:17:16.876269 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:16.876247    2584 patch_prober.go:28] interesting pod/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 29 21:17:16.876350 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:16.876291    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8" podUID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 29 21:17:17.875528 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:17.875493    2584 patch_prober.go:28] interesting pod/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 29 21:17:17.875922 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:17.875552    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8" podUID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 29 21:17:18.875051 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:18.875016    2584 patch_prober.go:28] interesting pod/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 29 21:17:18.875244 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:18.875073    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8" podUID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 29 21:17:19.875312 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:19.875270    2584 patch_prober.go:28] interesting pod/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 29 21:17:19.875677 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:19.875327    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8" podUID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 29 21:17:20.879486 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:20.879458    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:17:20.879856 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:20.879691    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:17:20.880288 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:20.880270    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:17:25.926408 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:25.926365    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw"]
Jul 29 21:17:25.926779 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:25.926670    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" containerName="istio-proxy"
Jul 29 21:17:25.926779 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:25.926681    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" containerName="istio-proxy"
Jul 29 21:17:25.926779 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:25.926758    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" containerName="istio-proxy"
Jul 29 21:17:25.929327 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:25.929310    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw"
Jul 29 21:17:25.931748 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:25.931722    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:17:25.931858 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:25.931734    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\""
Jul 29 21:17:25.936932 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:25.936909    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw"]
Jul 29 21:17:26.070685 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:26.070651    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d15de5-b76f-4c86-9155-e8553b5e1289-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw\" (UID: \"56d15de5-b76f-4c86-9155-e8553b5e1289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw"
Jul 29 21:17:26.171489 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:26.171455    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d15de5-b76f-4c86-9155-e8553b5e1289-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw\" (UID: \"56d15de5-b76f-4c86-9155-e8553b5e1289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw"
Jul 29 21:17:26.172086 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:26.172067    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d15de5-b76f-4c86-9155-e8553b5e1289-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw\" (UID: \"56d15de5-b76f-4c86-9155-e8553b5e1289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw"
Jul 29 21:17:26.241661 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:26.241587    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw"
Jul 29 21:17:26.367963 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:26.367938    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw"]
Jul 29 21:17:26.370422 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:17:26.370398    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d15de5_b76f_4c86_9155_e8553b5e1289.slice/crio-90c2738d7503ea5044f8504a0bbccd1c6f7e4b4fb4fd0fbbed1095e9965027a6 WatchSource:0}: Error finding container 90c2738d7503ea5044f8504a0bbccd1c6f7e4b4fb4fd0fbbed1095e9965027a6: Status 404 returned error can't find the container with id 90c2738d7503ea5044f8504a0bbccd1c6f7e4b4fb4fd0fbbed1095e9965027a6
Jul 29 21:17:26.586249 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:26.586175    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw" event={"ID":"56d15de5-b76f-4c86-9155-e8553b5e1289","Type":"ContainerStarted","Data":"90c2738d7503ea5044f8504a0bbccd1c6f7e4b4fb4fd0fbbed1095e9965027a6"}
Jul 29 21:17:27.597316 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:27.597280    2584 generic.go:356] "Generic (PLEG): container finished" podID="56d15de5-b76f-4c86-9155-e8553b5e1289" containerID="33ccc922060b92b8c40d3d3545c383fff7a2be8cbf38c610b89143bbe0f56e14" exitCode=0
Jul 29 21:17:27.597692 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:27.597323    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw" event={"ID":"56d15de5-b76f-4c86-9155-e8553b5e1289","Type":"ContainerDied","Data":"33ccc922060b92b8c40d3d3545c383fff7a2be8cbf38c610b89143bbe0f56e14"}
Jul 29 21:17:28.728769 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:28.728746    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw"
Jul 29 21:17:28.895452 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:28.895373    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d15de5-b76f-4c86-9155-e8553b5e1289-maas-api-service-ca\") pod \"56d15de5-b76f-4c86-9155-e8553b5e1289\" (UID: \"56d15de5-b76f-4c86-9155-e8553b5e1289\") "
Jul 29 21:17:28.895699 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:28.895677    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d15de5-b76f-4c86-9155-e8553b5e1289-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "56d15de5-b76f-4c86-9155-e8553b5e1289" (UID: "56d15de5-b76f-4c86-9155-e8553b5e1289"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:17:28.996725 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:28.996696    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d15de5-b76f-4c86-9155-e8553b5e1289-maas-api-service-ca\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:29.606516 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:29.606485    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw"
Jul 29 21:17:29.606692 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:29.606519    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw" event={"ID":"56d15de5-b76f-4c86-9155-e8553b5e1289","Type":"ContainerDied","Data":"90c2738d7503ea5044f8504a0bbccd1c6f7e4b4fb4fd0fbbed1095e9965027a6"}
Jul 29 21:17:29.606692 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:29.606557    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90c2738d7503ea5044f8504a0bbccd1c6f7e4b4fb4fd0fbbed1095e9965027a6"
Jul 29 21:17:32.568952 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:32.568901    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw"]
Jul 29 21:17:32.570517 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:32.570493    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-946d3187-nbmxw"]
Jul 29 21:17:32.706650 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:32.706618    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"]
Jul 29 21:17:32.706853 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:32.706830    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8" podUID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerName="istio-proxy" containerID="cri-o://02e7877412ebebc6c618637ddfaaf1f459d6f7b782b6532df3c73cf5cf2d15e3" gracePeriod=30
Jul 29 21:17:32.865058 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:32.865027    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56d15de5-b76f-4c86-9155-e8553b5e1289" path="/var/lib/kubelet/pods/56d15de5-b76f-4c86-9155-e8553b5e1289/volumes"
Jul 29 21:17:33.513108 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.513016    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"]
Jul 29 21:17:33.513736 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.513714    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d15de5-b76f-4c86-9155-e8553b5e1289" containerName="revoke-keys"
Jul 29 21:17:33.513736 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.513738    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d15de5-b76f-4c86-9155-e8553b5e1289" containerName="revoke-keys"
Jul 29 21:17:33.513903 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.513783    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" containerName="istio-proxy"
Jul 29 21:17:33.513903 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.513792    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" containerName="istio-proxy"
Jul 29 21:17:33.513995 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.513931    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d15de5-b76f-4c86-9155-e8553b5e1289" containerName="revoke-keys"
Jul 29 21:17:33.513995 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.513943    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81739c1-7295-4b75-9179-94dd48131d6b" containerName="istio-proxy"
Jul 29 21:17:33.518968 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.518949    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.521171 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.521133    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-17bc2b8b-openshift-default-dockercfg-pjs9c\""
Jul 29 21:17:33.527432 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.527406    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"]
Jul 29 21:17:33.538610 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.538582    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-credential-socket\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.538724 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.538614    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-podinfo\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.538724 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.538629    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-token\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.538724 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.538698    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istiod-ca-cert\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.538898 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.538742    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndb2v\" (UniqueName: \"kubernetes.io/projected/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-kube-api-access-ndb2v\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.538898 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.538761    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-data\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.538898 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.538785    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-workload-socket\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.538898 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.538826    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-workload-certs\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.538898 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.538841    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-envoy\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.639324 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.639290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-workload-certs\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.639324 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.639326    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-envoy\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.639836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.639365    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-credential-socket\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.639836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.639392    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-podinfo\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.639836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.639418    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-token\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.639836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.639465    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istiod-ca-cert\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.639836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.639506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndb2v\" (UniqueName: \"kubernetes.io/projected/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-kube-api-access-ndb2v\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.639836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.639534    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-data\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.639836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.639574    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-workload-socket\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.639836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.639711    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-workload-certs\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.639836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.639764    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-credential-socket\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.640201 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.639911    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-workload-socket\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.640201 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.639989    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-data\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.640277 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.640259    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istiod-ca-cert\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.642156 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.642113    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-envoy\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.642301 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.642284    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-podinfo\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.646890 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.646864    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-token\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.647393 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.647373    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndb2v\" (UniqueName: \"kubernetes.io/projected/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-kube-api-access-ndb2v\") pod \"e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") " pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.829853 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.829775    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:33.958457 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.958433    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"]
Jul 29 21:17:33.962039 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:17:33.962009    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e1ce3e_ce6c_4fd2_a32e_7b932b50d6a0.slice/crio-52041dfe6a1dae258e878ad25d09d2fd4ca65a825e5840f4abf779854c890d2b WatchSource:0}: Error finding container 52041dfe6a1dae258e878ad25d09d2fd4ca65a825e5840f4abf779854c890d2b: Status 404 returned error can't find the container with id 52041dfe6a1dae258e878ad25d09d2fd4ca65a825e5840f4abf779854c890d2b
Jul 29 21:17:33.964457 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.964425    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:17:33.964563 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.964504    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:17:33.964563 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:33.964550    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:17:34.630557 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:34.630523    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5" event={"ID":"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0","Type":"ContainerStarted","Data":"503dfcfcd82fa58f55bbc26889b0874236ed9620e2230945fd048f2aa5dfbfef"}
Jul 29 21:17:34.630799 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:34.630581    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5" event={"ID":"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0","Type":"ContainerStarted","Data":"52041dfe6a1dae258e878ad25d09d2fd4ca65a825e5840f4abf779854c890d2b"}
Jul 29 21:17:34.651522 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:34.651478    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5" podStartSLOduration=1.6514602109999998 podStartE2EDuration="1.651460211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:34.651021145 +0000 UTC m=+1870.247312840" watchObservedRunningTime="2026-07-29 21:17:34.651460211 +0000 UTC m=+1870.247751894"
Jul 29 21:17:34.829891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:34.829855    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:35.835050 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:35.835015    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:35.882129 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:35.882087    2584 patch_prober.go:28] interesting pod/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 29 21:17:35.882286 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:35.882156    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8" podUID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:17:36.641994 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:36.641958    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:36.645339 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:36.645316    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:37.951807 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.951781    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:17:37.976314 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976255    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-workload-socket\") pod \"2a25a3b1-6478-4651-85fb-6e157654acd9\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") "
Jul 29 21:17:37.976314 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976300    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a25a3b1-6478-4651-85fb-6e157654acd9-istiod-ca-cert\") pod \"2a25a3b1-6478-4651-85fb-6e157654acd9\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") "
Jul 29 21:17:37.976493 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976331    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-podinfo\") pod \"2a25a3b1-6478-4651-85fb-6e157654acd9\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") "
Jul 29 21:17:37.976493 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976375    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-workload-certs\") pod \"2a25a3b1-6478-4651-85fb-6e157654acd9\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") "
Jul 29 21:17:37.976493 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976398    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-envoy\") pod \"2a25a3b1-6478-4651-85fb-6e157654acd9\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") "
Jul 29 21:17:37.976493 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976433    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-token\") pod \"2a25a3b1-6478-4651-85fb-6e157654acd9\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") "
Jul 29 21:17:37.976493 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976465    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r69xb\" (UniqueName: \"kubernetes.io/projected/2a25a3b1-6478-4651-85fb-6e157654acd9-kube-api-access-r69xb\") pod \"2a25a3b1-6478-4651-85fb-6e157654acd9\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") "
Jul 29 21:17:37.976730 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976499    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-credential-socket\") pod \"2a25a3b1-6478-4651-85fb-6e157654acd9\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") "
Jul 29 21:17:37.976730 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976542    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-data\") pod \"2a25a3b1-6478-4651-85fb-6e157654acd9\" (UID: \"2a25a3b1-6478-4651-85fb-6e157654acd9\") "
Jul 29 21:17:37.976730 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976534    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a25a3b1-6478-4651-85fb-6e157654acd9" (UID: "2a25a3b1-6478-4651-85fb-6e157654acd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:17:37.976730 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976660    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a25a3b1-6478-4651-85fb-6e157654acd9" (UID: "2a25a3b1-6478-4651-85fb-6e157654acd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:17:37.976730 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976690    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a25a3b1-6478-4651-85fb-6e157654acd9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a25a3b1-6478-4651-85fb-6e157654acd9" (UID: "2a25a3b1-6478-4651-85fb-6e157654acd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:17:37.976982 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976816    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a25a3b1-6478-4651-85fb-6e157654acd9" (UID: "2a25a3b1-6478-4651-85fb-6e157654acd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:17:37.976982 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976952    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:37.976982 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976973    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:37.977128 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.976990    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a25a3b1-6478-4651-85fb-6e157654acd9-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:37.977128 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.977003    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:37.977467 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.977445    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a25a3b1-6478-4651-85fb-6e157654acd9" (UID: "2a25a3b1-6478-4651-85fb-6e157654acd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:17:37.979015 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.978989    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a25a3b1-6478-4651-85fb-6e157654acd9" (UID: "2a25a3b1-6478-4651-85fb-6e157654acd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:17:37.979206 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.979053    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a25a3b1-6478-4651-85fb-6e157654acd9-kube-api-access-r69xb" (OuterVolumeSpecName: "kube-api-access-r69xb") pod "2a25a3b1-6478-4651-85fb-6e157654acd9" (UID: "2a25a3b1-6478-4651-85fb-6e157654acd9"). InnerVolumeSpecName "kube-api-access-r69xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:17:37.979206 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.979063    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a25a3b1-6478-4651-85fb-6e157654acd9" (UID: "2a25a3b1-6478-4651-85fb-6e157654acd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:17:37.979387 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:37.979362    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a25a3b1-6478-4651-85fb-6e157654acd9" (UID: "2a25a3b1-6478-4651-85fb-6e157654acd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:17:38.077586 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.077556    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r69xb\" (UniqueName: \"kubernetes.io/projected/2a25a3b1-6478-4651-85fb-6e157654acd9-kube-api-access-r69xb\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:38.077586 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.077583    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:38.077586 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.077592    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:38.077821 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.077599    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:38.077821 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.077609    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a25a3b1-6478-4651-85fb-6e157654acd9-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:17:38.651269 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.651236    2584 generic.go:356] "Generic (PLEG): container finished" podID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerID="02e7877412ebebc6c618637ddfaaf1f459d6f7b782b6532df3c73cf5cf2d15e3" exitCode=0
Jul 29 21:17:38.651431 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.651291    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"
Jul 29 21:17:38.651431 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.651325    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8" event={"ID":"2a25a3b1-6478-4651-85fb-6e157654acd9","Type":"ContainerDied","Data":"02e7877412ebebc6c618637ddfaaf1f459d6f7b782b6532df3c73cf5cf2d15e3"}
Jul 29 21:17:38.651431 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.651383    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8" event={"ID":"2a25a3b1-6478-4651-85fb-6e157654acd9","Type":"ContainerDied","Data":"04709dee4ac5b02c6c56adca8b90138c0786ce7a057ea1253381becf6bd12489"}
Jul 29 21:17:38.651431 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.651415    2584 scope.go:117] "RemoveContainer" containerID="02e7877412ebebc6c618637ddfaaf1f459d6f7b782b6532df3c73cf5cf2d15e3"
Jul 29 21:17:38.660564 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.660548    2584 scope.go:117] "RemoveContainer" containerID="8af72b15ff433c9d826c9cc9cc67bab9a79780501e47b01594e61b225b4491c3"
Jul 29 21:17:38.667985 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.667967    2584 scope.go:117] "RemoveContainer" containerID="02e7877412ebebc6c618637ddfaaf1f459d6f7b782b6532df3c73cf5cf2d15e3"
Jul 29 21:17:38.668247 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:17:38.668230    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e7877412ebebc6c618637ddfaaf1f459d6f7b782b6532df3c73cf5cf2d15e3\": container with ID starting with 02e7877412ebebc6c618637ddfaaf1f459d6f7b782b6532df3c73cf5cf2d15e3 not found: ID does not exist" containerID="02e7877412ebebc6c618637ddfaaf1f459d6f7b782b6532df3c73cf5cf2d15e3"
Jul 29 21:17:38.668316 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.668255    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e7877412ebebc6c618637ddfaaf1f459d6f7b782b6532df3c73cf5cf2d15e3"} err="failed to get container status \"02e7877412ebebc6c618637ddfaaf1f459d6f7b782b6532df3c73cf5cf2d15e3\": rpc error: code = NotFound desc = could not find container \"02e7877412ebebc6c618637ddfaaf1f459d6f7b782b6532df3c73cf5cf2d15e3\": container with ID starting with 02e7877412ebebc6c618637ddfaaf1f459d6f7b782b6532df3c73cf5cf2d15e3 not found: ID does not exist"
Jul 29 21:17:38.668316 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.668271    2584 scope.go:117] "RemoveContainer" containerID="8af72b15ff433c9d826c9cc9cc67bab9a79780501e47b01594e61b225b4491c3"
Jul 29 21:17:38.668486 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:17:38.668467    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af72b15ff433c9d826c9cc9cc67bab9a79780501e47b01594e61b225b4491c3\": container with ID starting with 8af72b15ff433c9d826c9cc9cc67bab9a79780501e47b01594e61b225b4491c3 not found: ID does not exist" containerID="8af72b15ff433c9d826c9cc9cc67bab9a79780501e47b01594e61b225b4491c3"
Jul 29 21:17:38.668524 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.668492    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af72b15ff433c9d826c9cc9cc67bab9a79780501e47b01594e61b225b4491c3"} err="failed to get container status \"8af72b15ff433c9d826c9cc9cc67bab9a79780501e47b01594e61b225b4491c3\": rpc error: code = NotFound desc = could not find container \"8af72b15ff433c9d826c9cc9cc67bab9a79780501e47b01594e61b225b4491c3\": container with ID starting with 8af72b15ff433c9d826c9cc9cc67bab9a79780501e47b01594e61b225b4491c3 not found: ID does not exist"
Jul 29 21:17:38.671685 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.671665    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"]
Jul 29 21:17:38.678877 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.678857    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-946d3187-gw-openshift-default-5786d7b996-jlxc8"]
Jul 29 21:17:38.864756 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:38.864726    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a25a3b1-6478-4651-85fb-6e157654acd9" path="/var/lib/kubelet/pods/2a25a3b1-6478-4651-85fb-6e157654acd9/volumes"
Jul 29 21:17:50.707180 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:50.707124    2584 generic.go:356] "Generic (PLEG): container finished" podID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerID="503dfcfcd82fa58f55bbc26889b0874236ed9620e2230945fd048f2aa5dfbfef" exitCode=0
Jul 29 21:17:50.707180 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:50.707181    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5" event={"ID":"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0","Type":"ContainerDied","Data":"503dfcfcd82fa58f55bbc26889b0874236ed9620e2230945fd048f2aa5dfbfef"}
Jul 29 21:17:50.707611 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:50.707570    2584 scope.go:117] "RemoveContainer" containerID="503dfcfcd82fa58f55bbc26889b0874236ed9620e2230945fd048f2aa5dfbfef"
Jul 29 21:17:50.708583 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:50.708554    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:17:50.708634 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:50.708621    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:17:50.708667 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:50.708650    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:17:51.642787 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:51.642751    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:51.715458 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:51.715426    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5" event={"ID":"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0","Type":"ContainerStarted","Data":"a2a36e93a1a7298af0ecfe8aa7a504bc9e30fa6dc29a164aec6174f2517a42cf"}
Jul 29 21:17:51.829995 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:51.829953    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:51.831016 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:51.830991    2584 patch_prober.go:28] interesting pod/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 29 21:17:51.831130 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:51.831053    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5" podUID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 29 21:17:52.830418 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:52.830381    2584 patch_prober.go:28] interesting pod/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 29 21:17:52.830809 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:52.830445    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5" podUID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 29 21:17:53.830493 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:53.830459    2584 patch_prober.go:28] interesting pod/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 29 21:17:53.830857 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:53.830523    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5" podUID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 29 21:17:54.830524 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:54.830490    2584 patch_prober.go:28] interesting pod/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 29 21:17:54.830909 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:54.830548    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5" podUID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 29 21:17:55.834196 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:55.834164    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:55.834635 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:55.834430    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:17:55.835003 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:17:55.834985    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:18:01.741316 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.741277    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262"]
Jul 29 21:18:01.741683 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.741589    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerName="istio-proxy"
Jul 29 21:18:01.741683 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.741599    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerName="istio-proxy"
Jul 29 21:18:01.741683 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.741612    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerName="istio-proxy"
Jul 29 21:18:01.741683 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.741618    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerName="istio-proxy"
Jul 29 21:18:01.741683 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.741675    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerName="istio-proxy"
Jul 29 21:18:01.741683 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.741684    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a25a3b1-6478-4651-85fb-6e157654acd9" containerName="istio-proxy"
Jul 29 21:18:01.744520 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.744504    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262"
Jul 29 21:18:01.747876 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.746483    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\""
Jul 29 21:18:01.747876 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.746986    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:18:01.752427 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.752399    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262"]
Jul 29 21:18:01.890192 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.890160    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee98a89-8581-4f23-ba8b-208eb35690e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-17bc2b8b-52262\" (UID: \"0ee98a89-8581-4f23-ba8b-208eb35690e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262"
Jul 29 21:18:01.990751 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.990711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee98a89-8581-4f23-ba8b-208eb35690e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-17bc2b8b-52262\" (UID: \"0ee98a89-8581-4f23-ba8b-208eb35690e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262"
Jul 29 21:18:01.991573 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:01.991502    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee98a89-8581-4f23-ba8b-208eb35690e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-17bc2b8b-52262\" (UID: \"0ee98a89-8581-4f23-ba8b-208eb35690e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262"
Jul 29 21:18:02.058840 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:02.058805    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262"
Jul 29 21:18:02.190342 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:02.190316    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262"]
Jul 29 21:18:02.192795 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:18:02.192772    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee98a89_8581_4f23_ba8b_208eb35690e2.slice/crio-d7c4b6b3cc353d8d880caf08c24436243510e57c549d9085347673662d8ab00d WatchSource:0}: Error finding container d7c4b6b3cc353d8d880caf08c24436243510e57c549d9085347673662d8ab00d: Status 404 returned error can't find the container with id d7c4b6b3cc353d8d880caf08c24436243510e57c549d9085347673662d8ab00d
Jul 29 21:18:02.764949 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:02.764857    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262" event={"ID":"0ee98a89-8581-4f23-ba8b-208eb35690e2","Type":"ContainerStarted","Data":"d236fd3a7948049f9727dc94d126402632f1c08884feb0890afa88eff9f78ec0"}
Jul 29 21:18:02.764949 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:02.764914    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262" event={"ID":"0ee98a89-8581-4f23-ba8b-208eb35690e2","Type":"ContainerStarted","Data":"d7c4b6b3cc353d8d880caf08c24436243510e57c549d9085347673662d8ab00d"}
Jul 29 21:18:02.779887 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:02.779833    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262" podStartSLOduration=1.7798191270000001 podStartE2EDuration="1.779819127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:02.778247442 +0000 UTC m=+1898.374539152" watchObservedRunningTime="2026-07-29 21:18:02.779819127 +0000 UTC m=+1898.376110856"
Jul 29 21:18:03.771360 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:03.771329    2584 generic.go:356] "Generic (PLEG): container finished" podID="0ee98a89-8581-4f23-ba8b-208eb35690e2" containerID="d236fd3a7948049f9727dc94d126402632f1c08884feb0890afa88eff9f78ec0" exitCode=0
Jul 29 21:18:03.771738 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:03.771404    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262" event={"ID":"0ee98a89-8581-4f23-ba8b-208eb35690e2","Type":"ContainerDied","Data":"d236fd3a7948049f9727dc94d126402632f1c08884feb0890afa88eff9f78ec0"}
Jul 29 21:18:04.908156 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:04.908125    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262"
Jul 29 21:18:05.018666 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:05.018633    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee98a89-8581-4f23-ba8b-208eb35690e2-maas-api-service-ca\") pod \"0ee98a89-8581-4f23-ba8b-208eb35690e2\" (UID: \"0ee98a89-8581-4f23-ba8b-208eb35690e2\") "
Jul 29 21:18:05.018945 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:05.018918    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee98a89-8581-4f23-ba8b-208eb35690e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ee98a89-8581-4f23-ba8b-208eb35690e2" (UID: "0ee98a89-8581-4f23-ba8b-208eb35690e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:18:05.019017 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:05.019004    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee98a89-8581-4f23-ba8b-208eb35690e2-maas-api-service-ca\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:05.784438 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:05.784400    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262" event={"ID":"0ee98a89-8581-4f23-ba8b-208eb35690e2","Type":"ContainerDied","Data":"d7c4b6b3cc353d8d880caf08c24436243510e57c549d9085347673662d8ab00d"}
Jul 29 21:18:05.784601 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:05.784449    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7c4b6b3cc353d8d880caf08c24436243510e57c549d9085347673662d8ab00d"
Jul 29 21:18:05.784601 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:05.784413    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262"
Jul 29 21:18:07.968096 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:07.968059    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262"]
Jul 29 21:18:07.975530 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:07.975501    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17bc2b8b-52262"]
Jul 29 21:18:08.864815 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:08.864781    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ee98a89-8581-4f23-ba8b-208eb35690e2" path="/var/lib/kubelet/pods/0ee98a89-8581-4f23-ba8b-208eb35690e2/volumes"
Jul 29 21:18:09.078890 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.078858    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"]
Jul 29 21:18:09.079306 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.079065    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5" podUID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerName="istio-proxy" containerID="cri-o://a2a36e93a1a7298af0ecfe8aa7a504bc9e30fa6dc29a164aec6174f2517a42cf" gracePeriod=30
Jul 29 21:18:09.812558 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.812528    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"]
Jul 29 21:18:09.812869 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.812856    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee98a89-8581-4f23-ba8b-208eb35690e2" containerName="revoke-keys"
Jul 29 21:18:09.812939 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.812871    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee98a89-8581-4f23-ba8b-208eb35690e2" containerName="revoke-keys"
Jul 29 21:18:09.812976 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.812959    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee98a89-8581-4f23-ba8b-208eb35690e2" containerName="revoke-keys"
Jul 29 21:18:09.816263 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.816245    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:09.818072 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.818055    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2d0e6b56-openshift-default-dockercfg-4kb4x\""
Jul 29 21:18:09.826734 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.826711    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"]
Jul 29 21:18:09.964128 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.964088    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fd2c79-3dd0-431c-9c99-082093350701-istio-podinfo\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:09.964352 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.964169    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-workload-socket\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:09.964352 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.964186    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-istio-data\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:09.964352 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.964270    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk7qw\" (UniqueName: \"kubernetes.io/projected/44fd2c79-3dd0-431c-9c99-082093350701-kube-api-access-dk7qw\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:09.964352 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.964344    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-istio-envoy\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:09.964569 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.964378    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-workload-certs\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:09.964569 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.964458    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-credential-socket\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:09.964569 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.964484    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fd2c79-3dd0-431c-9c99-082093350701-istio-token\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:09.964569 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:09.964511    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fd2c79-3dd0-431c-9c99-082093350701-istiod-ca-cert\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.065443 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.065360    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-credential-socket\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.065443 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.065394    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fd2c79-3dd0-431c-9c99-082093350701-istio-token\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.065443 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.065413    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fd2c79-3dd0-431c-9c99-082093350701-istiod-ca-cert\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.065761 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.065453    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fd2c79-3dd0-431c-9c99-082093350701-istio-podinfo\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.065761 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.065491    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-workload-socket\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.065761 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.065512    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-istio-data\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.065761 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.065547    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk7qw\" (UniqueName: \"kubernetes.io/projected/44fd2c79-3dd0-431c-9c99-082093350701-kube-api-access-dk7qw\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.065761 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.065601    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-istio-envoy\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.065761 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.065631    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-workload-certs\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.066041 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.065848    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-credential-socket\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.066041 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.065879    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-workload-socket\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.066041 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.065955    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-istio-data\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.066041 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.065981    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-workload-certs\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.066297 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.066279    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fd2c79-3dd0-431c-9c99-082093350701-istiod-ca-cert\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.067920 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.067894    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-istio-envoy\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.068063 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.068020    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fd2c79-3dd0-431c-9c99-082093350701-istio-podinfo\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.072342 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.072322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fd2c79-3dd0-431c-9c99-082093350701-istio-token\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.072496 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.072480    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk7qw\" (UniqueName: \"kubernetes.io/projected/44fd2c79-3dd0-431c-9c99-082093350701-kube-api-access-dk7qw\") pod \"e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") " pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.126474 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.126446    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:10.257445 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.257418    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"]
Jul 29 21:18:10.259548 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:18:10.259519    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fd2c79_3dd0_431c_9c99_082093350701.slice/crio-8a76e72e52833adcc7597d518b9b384f561789233784f3ce5824f9cfa7357164 WatchSource:0}: Error finding container 8a76e72e52833adcc7597d518b9b384f561789233784f3ce5824f9cfa7357164: Status 404 returned error can't find the container with id 8a76e72e52833adcc7597d518b9b384f561789233784f3ce5824f9cfa7357164
Jul 29 21:18:10.261674 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.261645    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:18:10.261731 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.261709    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:18:10.261769 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.261743    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:18:10.810350 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.810316    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4" event={"ID":"44fd2c79-3dd0-431c-9c99-082093350701","Type":"ContainerStarted","Data":"86f290e664e0586df9affebdd05c5bb574d6df790aebd31bf5588fdf8e819fed"}
Jul 29 21:18:10.810522 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.810368    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4" event={"ID":"44fd2c79-3dd0-431c-9c99-082093350701","Type":"ContainerStarted","Data":"8a76e72e52833adcc7597d518b9b384f561789233784f3ce5824f9cfa7357164"}
Jul 29 21:18:10.829000 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.828953    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4" podStartSLOduration=1.828938649 podStartE2EDuration="1.828938649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:10.826760416 +0000 UTC m=+1906.423052101" watchObservedRunningTime="2026-07-29 21:18:10.828938649 +0000 UTC m=+1906.425230333"
Jul 29 21:18:10.837014 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.836985    2584 patch_prober.go:28] interesting pod/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 21:18:10.837118 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:10.837038    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5" podUID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:18:11.127517 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:11.127485    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:11.132108 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:11.132082    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:11.815404 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:11.815371    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:11.816301 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:11.816282    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:14.331404 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.331383    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:18:14.504108 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504076    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-data\") pod \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") "
Jul 29 21:18:14.504310 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504114    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-podinfo\") pod \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") "
Jul 29 21:18:14.504310 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504197    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-workload-socket\") pod \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") "
Jul 29 21:18:14.504310 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504221    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-envoy\") pod \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") "
Jul 29 21:18:14.504310 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504252    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndb2v\" (UniqueName: \"kubernetes.io/projected/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-kube-api-access-ndb2v\") pod \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") "
Jul 29 21:18:14.504310 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504302    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istiod-ca-cert\") pod \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") "
Jul 29 21:18:14.504631 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504373    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-token\") pod \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") "
Jul 29 21:18:14.504631 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504402    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-credential-socket\") pod \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") "
Jul 29 21:18:14.504631 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504430    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-workload-certs\") pod \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\" (UID: \"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0\") "
Jul 29 21:18:14.504631 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504493    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" (UID: "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:18:14.504837 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504691    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" (UID: "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:18:14.504837 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504721    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:14.504837 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504757    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" (UID: "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:18:14.504837 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504764    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" (UID: "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:18:14.505036 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.504948    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-data" (OuterVolumeSpecName: "istio-data") pod "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" (UID: "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:18:14.506619 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.506592    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" (UID: "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:18:14.506733 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.506598    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" (UID: "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:18:14.506733 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.506725    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" (UID: "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:18:14.506843 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.506751    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-kube-api-access-ndb2v" (OuterVolumeSpecName: "kube-api-access-ndb2v") pod "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" (UID: "04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0"). InnerVolumeSpecName "kube-api-access-ndb2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:18:14.605212 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.605187    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:14.605212 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.605209    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:14.605212 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.605218    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:14.605410 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.605230    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:14.605410 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.605239    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:14.605410 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.605246    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:14.605410 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.605253    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:14.605410 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.605262    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndb2v\" (UniqueName: \"kubernetes.io/projected/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0-kube-api-access-ndb2v\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:14.832308 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.832217    2584 generic.go:356] "Generic (PLEG): container finished" podID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerID="a2a36e93a1a7298af0ecfe8aa7a504bc9e30fa6dc29a164aec6174f2517a42cf" exitCode=0
Jul 29 21:18:14.832308 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.832280    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"
Jul 29 21:18:14.832528 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.832283    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5" event={"ID":"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0","Type":"ContainerDied","Data":"a2a36e93a1a7298af0ecfe8aa7a504bc9e30fa6dc29a164aec6174f2517a42cf"}
Jul 29 21:18:14.832528 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.832409    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5" event={"ID":"04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0","Type":"ContainerDied","Data":"52041dfe6a1dae258e878ad25d09d2fd4ca65a825e5840f4abf779854c890d2b"}
Jul 29 21:18:14.832528 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.832433    2584 scope.go:117] "RemoveContainer" containerID="a2a36e93a1a7298af0ecfe8aa7a504bc9e30fa6dc29a164aec6174f2517a42cf"
Jul 29 21:18:14.842237 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.842042    2584 scope.go:117] "RemoveContainer" containerID="503dfcfcd82fa58f55bbc26889b0874236ed9620e2230945fd048f2aa5dfbfef"
Jul 29 21:18:14.849922 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.849896    2584 scope.go:117] "RemoveContainer" containerID="a2a36e93a1a7298af0ecfe8aa7a504bc9e30fa6dc29a164aec6174f2517a42cf"
Jul 29 21:18:14.850289 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:18:14.850262    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a36e93a1a7298af0ecfe8aa7a504bc9e30fa6dc29a164aec6174f2517a42cf\": container with ID starting with a2a36e93a1a7298af0ecfe8aa7a504bc9e30fa6dc29a164aec6174f2517a42cf not found: ID does not exist" containerID="a2a36e93a1a7298af0ecfe8aa7a504bc9e30fa6dc29a164aec6174f2517a42cf"
Jul 29 21:18:14.850395 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.850296    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a36e93a1a7298af0ecfe8aa7a504bc9e30fa6dc29a164aec6174f2517a42cf"} err="failed to get container status \"a2a36e93a1a7298af0ecfe8aa7a504bc9e30fa6dc29a164aec6174f2517a42cf\": rpc error: code = NotFound desc = could not find container \"a2a36e93a1a7298af0ecfe8aa7a504bc9e30fa6dc29a164aec6174f2517a42cf\": container with ID starting with a2a36e93a1a7298af0ecfe8aa7a504bc9e30fa6dc29a164aec6174f2517a42cf not found: ID does not exist"
Jul 29 21:18:14.850395 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.850315    2584 scope.go:117] "RemoveContainer" containerID="503dfcfcd82fa58f55bbc26889b0874236ed9620e2230945fd048f2aa5dfbfef"
Jul 29 21:18:14.850567 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:18:14.850543    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503dfcfcd82fa58f55bbc26889b0874236ed9620e2230945fd048f2aa5dfbfef\": container with ID starting with 503dfcfcd82fa58f55bbc26889b0874236ed9620e2230945fd048f2aa5dfbfef not found: ID does not exist" containerID="503dfcfcd82fa58f55bbc26889b0874236ed9620e2230945fd048f2aa5dfbfef"
Jul 29 21:18:14.850681 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.850575    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503dfcfcd82fa58f55bbc26889b0874236ed9620e2230945fd048f2aa5dfbfef"} err="failed to get container status \"503dfcfcd82fa58f55bbc26889b0874236ed9620e2230945fd048f2aa5dfbfef\": rpc error: code = NotFound desc = could not find container \"503dfcfcd82fa58f55bbc26889b0874236ed9620e2230945fd048f2aa5dfbfef\": container with ID starting with 503dfcfcd82fa58f55bbc26889b0874236ed9620e2230945fd048f2aa5dfbfef not found: ID does not exist"
Jul 29 21:18:14.852383 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.852363    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"]
Jul 29 21:18:14.857826 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.857804    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-17bc2b8b-openshift-default-fb9b4594-tfds5"]
Jul 29 21:18:14.867160 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:14.867120    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" path="/var/lib/kubelet/pods/04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0/volumes"
Jul 29 21:18:25.880255 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:25.880221    2584 generic.go:356] "Generic (PLEG): container finished" podID="44fd2c79-3dd0-431c-9c99-082093350701" containerID="86f290e664e0586df9affebdd05c5bb574d6df790aebd31bf5588fdf8e819fed" exitCode=0
Jul 29 21:18:25.880683 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:25.880279    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4" event={"ID":"44fd2c79-3dd0-431c-9c99-082093350701","Type":"ContainerDied","Data":"86f290e664e0586df9affebdd05c5bb574d6df790aebd31bf5588fdf8e819fed"}
Jul 29 21:18:25.880683 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:25.880645    2584 scope.go:117] "RemoveContainer" containerID="86f290e664e0586df9affebdd05c5bb574d6df790aebd31bf5588fdf8e819fed"
Jul 29 21:18:25.881586 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:25.881559    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:18:25.881654 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:25.881642    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:18:25.881697 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:25.881676    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:18:26.816029 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:26.815990    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:26.886093 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:26.886060    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4" event={"ID":"44fd2c79-3dd0-431c-9c99-082093350701","Type":"ContainerStarted","Data":"69aa73bb1aaf0c4f90018d647096db709a15ae2750462de2de39221330d67d84"}
Jul 29 21:18:27.127305 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:27.127261    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:27.128358 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:27.128330    2584 patch_prober.go:28] interesting pod/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 29 21:18:27.128499 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:27.128396    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4" podUID="44fd2c79-3dd0-431c-9c99-082093350701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 29 21:18:28.127632 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:28.127590    2584 patch_prober.go:28] interesting pod/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 29 21:18:28.128020 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:28.127652    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4" podUID="44fd2c79-3dd0-431c-9c99-082093350701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 29 21:18:29.127449 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:29.127418    2584 patch_prober.go:28] interesting pod/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 29 21:18:29.127625 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:29.127474    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4" podUID="44fd2c79-3dd0-431c-9c99-082093350701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 29 21:18:30.127380 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:30.127339    2584 patch_prober.go:28] interesting pod/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 29 21:18:30.127766 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:30.127401    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4" podUID="44fd2c79-3dd0-431c-9c99-082093350701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 29 21:18:31.130707 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:31.130676    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:31.131097 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:31.130929    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:31.131607 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:31.131591    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:37.099200 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.099162    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z"]
Jul 29 21:18:37.099753 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.099479    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerName="istio-proxy"
Jul 29 21:18:37.099753 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.099489    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerName="istio-proxy"
Jul 29 21:18:37.099753 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.099498    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerName="istio-proxy"
Jul 29 21:18:37.099753 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.099504    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerName="istio-proxy"
Jul 29 21:18:37.099753 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.099568    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerName="istio-proxy"
Jul 29 21:18:37.099753 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.099576    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e1ce3e-ce6c-4fd2-a32e-7b932b50d6a0" containerName="istio-proxy"
Jul 29 21:18:37.102527 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.102507    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z"
Jul 29 21:18:37.104314 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.104289    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\""
Jul 29 21:18:37.104447 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.104393    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:18:37.111630 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.111606    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z"]
Jul 29 21:18:37.189987 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.189948    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d576c83-555d-463d-881a-6051e4ae4b54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z\" (UID: \"4d576c83-555d-463d-881a-6051e4ae4b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z"
Jul 29 21:18:37.290467 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.290430    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d576c83-555d-463d-881a-6051e4ae4b54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z\" (UID: \"4d576c83-555d-463d-881a-6051e4ae4b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z"
Jul 29 21:18:37.291024 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.291006    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d576c83-555d-463d-881a-6051e4ae4b54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z\" (UID: \"4d576c83-555d-463d-881a-6051e4ae4b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z"
Jul 29 21:18:37.414743 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.414654    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z"
Jul 29 21:18:37.539919 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.539887    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z"]
Jul 29 21:18:37.543572 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:18:37.543542    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d576c83_555d_463d_881a_6051e4ae4b54.slice/crio-52e86121c7749956dc43346578e3e3cc7a2b5f0b643c380fd6c818df2f6ee7f4 WatchSource:0}: Error finding container 52e86121c7749956dc43346578e3e3cc7a2b5f0b643c380fd6c818df2f6ee7f4: Status 404 returned error can't find the container with id 52e86121c7749956dc43346578e3e3cc7a2b5f0b643c380fd6c818df2f6ee7f4
Jul 29 21:18:37.942002 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.941960    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z" event={"ID":"4d576c83-555d-463d-881a-6051e4ae4b54","Type":"ContainerStarted","Data":"644b66cc201f7dbd0c15d7baf5669c18801f1e2968cf6a2f56c7335d29a97c0f"}
Jul 29 21:18:37.942203 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.942022    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z" event={"ID":"4d576c83-555d-463d-881a-6051e4ae4b54","Type":"ContainerStarted","Data":"52e86121c7749956dc43346578e3e3cc7a2b5f0b643c380fd6c818df2f6ee7f4"}
Jul 29 21:18:37.956090 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:37.956040    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z" podStartSLOduration=0.95602499 podStartE2EDuration="956.02499ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:37.954249675 +0000 UTC m=+1933.550541359" watchObservedRunningTime="2026-07-29 21:18:37.95602499 +0000 UTC m=+1933.552316704"
Jul 29 21:18:38.950276 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:38.950241    2584 generic.go:356] "Generic (PLEG): container finished" podID="4d576c83-555d-463d-881a-6051e4ae4b54" containerID="644b66cc201f7dbd0c15d7baf5669c18801f1e2968cf6a2f56c7335d29a97c0f" exitCode=0
Jul 29 21:18:38.950665 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:38.950328    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z" event={"ID":"4d576c83-555d-463d-881a-6051e4ae4b54","Type":"ContainerDied","Data":"644b66cc201f7dbd0c15d7baf5669c18801f1e2968cf6a2f56c7335d29a97c0f"}
Jul 29 21:18:40.085907 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:40.085882    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z"
Jul 29 21:18:40.116088 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:40.116062    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d576c83-555d-463d-881a-6051e4ae4b54-maas-api-service-ca\") pod \"4d576c83-555d-463d-881a-6051e4ae4b54\" (UID: \"4d576c83-555d-463d-881a-6051e4ae4b54\") "
Jul 29 21:18:40.116391 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:40.116370    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d576c83-555d-463d-881a-6051e4ae4b54-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4d576c83-555d-463d-881a-6051e4ae4b54" (UID: "4d576c83-555d-463d-881a-6051e4ae4b54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:18:40.216879 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:40.216799    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d576c83-555d-463d-881a-6051e4ae4b54-maas-api-service-ca\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:40.959552 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:40.959523    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z"
Jul 29 21:18:40.959552 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:40.959548    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z" event={"ID":"4d576c83-555d-463d-881a-6051e4ae4b54","Type":"ContainerDied","Data":"52e86121c7749956dc43346578e3e3cc7a2b5f0b643c380fd6c818df2f6ee7f4"}
Jul 29 21:18:40.959801 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:40.959588    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52e86121c7749956dc43346578e3e3cc7a2b5f0b643c380fd6c818df2f6ee7f4"
Jul 29 21:18:43.899571 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:43.899537    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z"]
Jul 29 21:18:43.901998 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:43.901977    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d0e6b56-2fx7z"]
Jul 29 21:18:44.043353 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:44.043321    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"]
Jul 29 21:18:44.043563 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:44.043537    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4" podUID="44fd2c79-3dd0-431c-9c99-082093350701" containerName="istio-proxy" containerID="cri-o://69aa73bb1aaf0c4f90018d647096db709a15ae2750462de2de39221330d67d84" gracePeriod=30
Jul 29 21:18:44.865128 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:44.865096    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d576c83-555d-463d-881a-6051e4ae4b54" path="/var/lib/kubelet/pods/4d576c83-555d-463d-881a-6051e4ae4b54/volumes"
Jul 29 21:18:46.132903 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:46.132871    2584 patch_prober.go:28] interesting pod/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 21:18:46.133292 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:46.132922    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4" podUID="44fd2c79-3dd0-431c-9c99-082093350701" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:18:49.294965 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.294936    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:49.391657 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.391624    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-istio-data\") pod \"44fd2c79-3dd0-431c-9c99-082093350701\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") "
Jul 29 21:18:49.391843 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.391668    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk7qw\" (UniqueName: \"kubernetes.io/projected/44fd2c79-3dd0-431c-9c99-082093350701-kube-api-access-dk7qw\") pod \"44fd2c79-3dd0-431c-9c99-082093350701\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") "
Jul 29 21:18:49.391843 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.391702    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-istio-envoy\") pod \"44fd2c79-3dd0-431c-9c99-082093350701\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") "
Jul 29 21:18:49.391843 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.391731    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-workload-socket\") pod \"44fd2c79-3dd0-431c-9c99-082093350701\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") "
Jul 29 21:18:49.392016 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.391877    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-workload-certs\") pod \"44fd2c79-3dd0-431c-9c99-082093350701\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") "
Jul 29 21:18:49.392016 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.391937    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fd2c79-3dd0-431c-9c99-082093350701-istiod-ca-cert\") pod \"44fd2c79-3dd0-431c-9c99-082093350701\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") "
Jul 29 21:18:49.392016 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.392003    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "44fd2c79-3dd0-431c-9c99-082093350701" (UID: "44fd2c79-3dd0-431c-9c99-082093350701"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:18:49.392189 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.392019    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fd2c79-3dd0-431c-9c99-082093350701-istio-podinfo\") pod \"44fd2c79-3dd0-431c-9c99-082093350701\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") "
Jul 29 21:18:49.392189 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.392051    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-credential-socket\") pod \"44fd2c79-3dd0-431c-9c99-082093350701\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") "
Jul 29 21:18:49.392189 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.392098    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fd2c79-3dd0-431c-9c99-082093350701-istio-token\") pod \"44fd2c79-3dd0-431c-9c99-082093350701\" (UID: \"44fd2c79-3dd0-431c-9c99-082093350701\") "
Jul 29 21:18:49.392347 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.392302    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "44fd2c79-3dd0-431c-9c99-082093350701" (UID: "44fd2c79-3dd0-431c-9c99-082093350701"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:18:49.392397 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.392355    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44fd2c79-3dd0-431c-9c99-082093350701-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "44fd2c79-3dd0-431c-9c99-082093350701" (UID: "44fd2c79-3dd0-431c-9c99-082093350701"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:18:49.392454 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.392432    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:49.392508 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.392452    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:49.392557 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.392522    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "44fd2c79-3dd0-431c-9c99-082093350701" (UID: "44fd2c79-3dd0-431c-9c99-082093350701"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:18:49.392610 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.392584    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-istio-data" (OuterVolumeSpecName: "istio-data") pod "44fd2c79-3dd0-431c-9c99-082093350701" (UID: "44fd2c79-3dd0-431c-9c99-082093350701"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:18:49.394570 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.394537    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44fd2c79-3dd0-431c-9c99-082093350701-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "44fd2c79-3dd0-431c-9c99-082093350701" (UID: "44fd2c79-3dd0-431c-9c99-082093350701"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:18:49.394570 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.394552    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fd2c79-3dd0-431c-9c99-082093350701-kube-api-access-dk7qw" (OuterVolumeSpecName: "kube-api-access-dk7qw") pod "44fd2c79-3dd0-431c-9c99-082093350701" (UID: "44fd2c79-3dd0-431c-9c99-082093350701"). InnerVolumeSpecName "kube-api-access-dk7qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:18:49.394767 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.394565    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fd2c79-3dd0-431c-9c99-082093350701-istio-token" (OuterVolumeSpecName: "istio-token") pod "44fd2c79-3dd0-431c-9c99-082093350701" (UID: "44fd2c79-3dd0-431c-9c99-082093350701"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:18:49.394767 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.394638    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "44fd2c79-3dd0-431c-9c99-082093350701" (UID: "44fd2c79-3dd0-431c-9c99-082093350701"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:18:49.493357 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.493322    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fd2c79-3dd0-431c-9c99-082093350701-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:49.493357 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.493349    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:49.493357 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.493360    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk7qw\" (UniqueName: \"kubernetes.io/projected/44fd2c79-3dd0-431c-9c99-082093350701-kube-api-access-dk7qw\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:49.493357 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.493368    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:49.493606 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.493377    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fd2c79-3dd0-431c-9c99-082093350701-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:49.493606 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.493385    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fd2c79-3dd0-431c-9c99-082093350701-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:49.493606 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:49.493392    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fd2c79-3dd0-431c-9c99-082093350701-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:18:50.001205 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:50.001172    2584 generic.go:356] "Generic (PLEG): container finished" podID="44fd2c79-3dd0-431c-9c99-082093350701" containerID="69aa73bb1aaf0c4f90018d647096db709a15ae2750462de2de39221330d67d84" exitCode=0
Jul 29 21:18:50.001353 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:50.001272    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"
Jul 29 21:18:50.001353 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:50.001282    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4" event={"ID":"44fd2c79-3dd0-431c-9c99-082093350701","Type":"ContainerDied","Data":"69aa73bb1aaf0c4f90018d647096db709a15ae2750462de2de39221330d67d84"}
Jul 29 21:18:50.001353 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:50.001327    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4" event={"ID":"44fd2c79-3dd0-431c-9c99-082093350701","Type":"ContainerDied","Data":"8a76e72e52833adcc7597d518b9b384f561789233784f3ce5824f9cfa7357164"}
Jul 29 21:18:50.001353 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:50.001348    2584 scope.go:117] "RemoveContainer" containerID="69aa73bb1aaf0c4f90018d647096db709a15ae2750462de2de39221330d67d84"
Jul 29 21:18:50.010762 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:50.010743    2584 scope.go:117] "RemoveContainer" containerID="86f290e664e0586df9affebdd05c5bb574d6df790aebd31bf5588fdf8e819fed"
Jul 29 21:18:50.018886 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:50.018868    2584 scope.go:117] "RemoveContainer" containerID="69aa73bb1aaf0c4f90018d647096db709a15ae2750462de2de39221330d67d84"
Jul 29 21:18:50.019125 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:18:50.019106    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69aa73bb1aaf0c4f90018d647096db709a15ae2750462de2de39221330d67d84\": container with ID starting with 69aa73bb1aaf0c4f90018d647096db709a15ae2750462de2de39221330d67d84 not found: ID does not exist" containerID="69aa73bb1aaf0c4f90018d647096db709a15ae2750462de2de39221330d67d84"
Jul 29 21:18:50.019223 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:50.019137    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69aa73bb1aaf0c4f90018d647096db709a15ae2750462de2de39221330d67d84"} err="failed to get container status \"69aa73bb1aaf0c4f90018d647096db709a15ae2750462de2de39221330d67d84\": rpc error: code = NotFound desc = could not find container \"69aa73bb1aaf0c4f90018d647096db709a15ae2750462de2de39221330d67d84\": container with ID starting with 69aa73bb1aaf0c4f90018d647096db709a15ae2750462de2de39221330d67d84 not found: ID does not exist"
Jul 29 21:18:50.019223 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:50.019178    2584 scope.go:117] "RemoveContainer" containerID="86f290e664e0586df9affebdd05c5bb574d6df790aebd31bf5588fdf8e819fed"
Jul 29 21:18:50.019436 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:18:50.019418    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f290e664e0586df9affebdd05c5bb574d6df790aebd31bf5588fdf8e819fed\": container with ID starting with 86f290e664e0586df9affebdd05c5bb574d6df790aebd31bf5588fdf8e819fed not found: ID does not exist" containerID="86f290e664e0586df9affebdd05c5bb574d6df790aebd31bf5588fdf8e819fed"
Jul 29 21:18:50.019509 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:50.019448    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f290e664e0586df9affebdd05c5bb574d6df790aebd31bf5588fdf8e819fed"} err="failed to get container status \"86f290e664e0586df9affebdd05c5bb574d6df790aebd31bf5588fdf8e819fed\": rpc error: code = NotFound desc = could not find container \"86f290e664e0586df9affebdd05c5bb574d6df790aebd31bf5588fdf8e819fed\": container with ID starting with 86f290e664e0586df9affebdd05c5bb574d6df790aebd31bf5588fdf8e819fed not found: ID does not exist"
Jul 29 21:18:50.022362 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:50.022343    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"]
Jul 29 21:18:50.027463 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:50.027444    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2d0e6b56-openshift-default-7b595bb5b4-77qp4"]
Jul 29 21:18:50.865514 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:18:50.865482    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44fd2c79-3dd0-431c-9c99-082093350701" path="/var/lib/kubelet/pods/44fd2c79-3dd0-431c-9c99-082093350701/volumes"
Jul 29 21:20:36.715075 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.715044    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"]
Jul 29 21:20:36.715506 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.715370    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44fd2c79-3dd0-431c-9c99-082093350701" containerName="istio-proxy"
Jul 29 21:20:36.715506 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.715382    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="44fd2c79-3dd0-431c-9c99-082093350701" containerName="istio-proxy"
Jul 29 21:20:36.715506 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.715395    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44fd2c79-3dd0-431c-9c99-082093350701" containerName="istio-proxy"
Jul 29 21:20:36.715506 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.715401    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="44fd2c79-3dd0-431c-9c99-082093350701" containerName="istio-proxy"
Jul 29 21:20:36.715506 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.715414    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d576c83-555d-463d-881a-6051e4ae4b54" containerName="revoke-keys"
Jul 29 21:20:36.715506 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.715420    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d576c83-555d-463d-881a-6051e4ae4b54" containerName="revoke-keys"
Jul 29 21:20:36.715506 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.715467    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d576c83-555d-463d-881a-6051e4ae4b54" containerName="revoke-keys"
Jul 29 21:20:36.715506 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.715475    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="44fd2c79-3dd0-431c-9c99-082093350701" containerName="istio-proxy"
Jul 29 21:20:36.715506 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.715482    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="44fd2c79-3dd0-431c-9c99-082093350701" containerName="istio-proxy"
Jul 29 21:20:36.718285 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.718268    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.720282 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.720259    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f7c67e2c-openshift-default-dockercfg-2lmhv\""
Jul 29 21:20:36.742771 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.742747    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"]
Jul 29 21:20:36.760128 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.760092    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b20990f-2284-4c68-a65f-add383f091e3-istio-token\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.760128 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.760120    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhv66\" (UniqueName: \"kubernetes.io/projected/9b20990f-2284-4c68-a65f-add383f091e3-kube-api-access-vhv66\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.760364 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.760172    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-credential-socket\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.760364 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.760202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b20990f-2284-4c68-a65f-add383f091e3-istio-podinfo\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.760364 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.760237    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-istio-data\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.760364 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.760258    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-istio-envoy\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.760364 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.760275    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-workload-certs\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.760636 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.760368    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-workload-socket\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.760636 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.760403    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b20990f-2284-4c68-a65f-add383f091e3-istiod-ca-cert\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.861191 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.861138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-workload-socket\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.861191 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.861197    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b20990f-2284-4c68-a65f-add383f091e3-istiod-ca-cert\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.861431 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.861233    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b20990f-2284-4c68-a65f-add383f091e3-istio-token\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.861431 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.861253    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhv66\" (UniqueName: \"kubernetes.io/projected/9b20990f-2284-4c68-a65f-add383f091e3-kube-api-access-vhv66\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.861431 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.861286    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-credential-socket\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.861431 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.861304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b20990f-2284-4c68-a65f-add383f091e3-istio-podinfo\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.861431 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.861328    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-istio-data\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.861757 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.861478    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-workload-socket\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.861757 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.861519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-istio-envoy\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.861757 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.861573    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-workload-certs\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.861917 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.861785    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-credential-socket\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.861917 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.861875    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-workload-certs\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.862008 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.861988    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-istio-data\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.862065 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.862047    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b20990f-2284-4c68-a65f-add383f091e3-istiod-ca-cert\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.863925 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.863900    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-istio-envoy\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.864082 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.864063    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b20990f-2284-4c68-a65f-add383f091e3-istio-podinfo\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.868743 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.868723    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b20990f-2284-4c68-a65f-add383f091e3-istio-token\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:36.868869 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:36.868851    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhv66\" (UniqueName: \"kubernetes.io/projected/9b20990f-2284-4c68-a65f-add383f091e3-kube-api-access-vhv66\") pod \"e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") " pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:37.028969 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:37.028892    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:37.162732 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:37.162708    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"]
Jul 29 21:20:37.165424 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:20:37.165397    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b20990f_2284_4c68_a65f_add383f091e3.slice/crio-c640f3ea2b98b4e33f70166d7b55d0ee53c29d893ef23c1674cdd9243a616504 WatchSource:0}: Error finding container c640f3ea2b98b4e33f70166d7b55d0ee53c29d893ef23c1674cdd9243a616504: Status 404 returned error can't find the container with id c640f3ea2b98b4e33f70166d7b55d0ee53c29d893ef23c1674cdd9243a616504
Jul 29 21:20:37.167619 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:37.167583    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:20:37.167724 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:37.167663    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:20:37.167724 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:37.167705    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:20:37.475498 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:37.475463    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq" event={"ID":"9b20990f-2284-4c68-a65f-add383f091e3","Type":"ContainerStarted","Data":"f087496413cb212f0d4b9131b0e5a602e76ee644387681777456f1bc986f4f5a"}
Jul 29 21:20:37.475679 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:37.475566    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq" event={"ID":"9b20990f-2284-4c68-a65f-add383f091e3","Type":"ContainerStarted","Data":"c640f3ea2b98b4e33f70166d7b55d0ee53c29d893ef23c1674cdd9243a616504"}
Jul 29 21:20:37.496181 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:37.496114    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq" podStartSLOduration=1.496099755 podStartE2EDuration="1.496099755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:20:37.49429156 +0000 UTC m=+2053.090583258" watchObservedRunningTime="2026-07-29 21:20:37.496099755 +0000 UTC m=+2053.092391439"
Jul 29 21:20:38.030007 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:38.029974    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:38.034511 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:38.034485    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:38.479742 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:38.479711    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:38.480654 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:38.480631    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:53.549600 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:53.548395    2584 patch_prober.go:28] interesting pod/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:36356->10.133.0.43:15021: read: connection reset by peer" start-of-body=
Jul 29 21:20:53.549600 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:53.548493    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq" podUID="9b20990f-2284-4c68-a65f-add383f091e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:36356->10.133.0.43:15021: read: connection reset by peer"
Jul 29 21:20:54.559990 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:54.559953    2584 generic.go:356] "Generic (PLEG): container finished" podID="9b20990f-2284-4c68-a65f-add383f091e3" containerID="f087496413cb212f0d4b9131b0e5a602e76ee644387681777456f1bc986f4f5a" exitCode=0
Jul 29 21:20:54.560387 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:54.560024    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq" event={"ID":"9b20990f-2284-4c68-a65f-add383f091e3","Type":"ContainerDied","Data":"f087496413cb212f0d4b9131b0e5a602e76ee644387681777456f1bc986f4f5a"}
Jul 29 21:20:54.560435 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:54.560414    2584 scope.go:117] "RemoveContainer" containerID="f087496413cb212f0d4b9131b0e5a602e76ee644387681777456f1bc986f4f5a"
Jul 29 21:20:54.561460 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:54.561434    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:20:54.561536 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:54.561517    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:20:54.561605 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:54.561591    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:20:55.565860 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:55.565820    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq" event={"ID":"9b20990f-2284-4c68-a65f-add383f091e3","Type":"ContainerStarted","Data":"eb5284fe699ea0399d36620b062116ec90a17de7d783b31f9cb2d8e620961cc0"}
Jul 29 21:20:56.029650 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:56.029603    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:20:56.029906 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:56.029883    2584 patch_prober.go:28] interesting pod/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 29 21:20:56.029992 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:56.029939    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq" podUID="9b20990f-2284-4c68-a65f-add383f091e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 29 21:20:57.029744 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:57.029706    2584 patch_prober.go:28] interesting pod/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 29 21:20:57.030169 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:57.029764    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq" podUID="9b20990f-2284-4c68-a65f-add383f091e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 29 21:20:58.029585 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:58.029547    2584 patch_prober.go:28] interesting pod/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 29 21:20:58.029770 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:58.029605    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq" podUID="9b20990f-2284-4c68-a65f-add383f091e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 29 21:20:59.029903 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:59.029866    2584 patch_prober.go:28] interesting pod/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 29 21:20:59.030342 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:20:59.029928    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq" podUID="9b20990f-2284-4c68-a65f-add383f091e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 29 21:21:00.032584 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:00.032551    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:21:00.033027 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:00.032943    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:21:00.033710 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:00.033692    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:21:16.228686 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.228651    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"]
Jul 29 21:21:16.232135 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.232114    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.234360 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.234338    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-58463a07-openshift-default-dockercfg-69wx7\""
Jul 29 21:21:16.245496 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.245475    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"]
Jul 29 21:21:16.291411 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.291380    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-workload-certs\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.291571 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.291418    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istiod-ca-cert\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.291571 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.291471    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-podinfo\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.291571 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.291542    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-token\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.291680 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.291580    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-workload-socket\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.291680 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.291643    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q47dt\" (UniqueName: \"kubernetes.io/projected/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-kube-api-access-q47dt\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.291680 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.291664    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-envoy\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.291777 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.291684    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-credential-socket\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.291777 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.291705    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-data\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.392274 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.392242    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q47dt\" (UniqueName: \"kubernetes.io/projected/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-kube-api-access-q47dt\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.392274 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.392277    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-envoy\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.392541 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.392298    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-credential-socket\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.392541 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.392314    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-data\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.392541 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.392331    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-workload-certs\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.392541 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.392354    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istiod-ca-cert\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.392541 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.392432    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-podinfo\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.392541 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.392476    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-token\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.392541 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.392500    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-workload-socket\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.392905 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.392755    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-credential-socket\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.392905 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.392795    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-workload-certs\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.393006 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.392950    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-workload-socket\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.393062 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.393001    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-data\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.393155 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.393118    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istiod-ca-cert\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.394812 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.394785    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-envoy\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.394965 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.394950    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-podinfo\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.400000 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.399971    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-token\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.400192 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.400171    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q47dt\" (UniqueName: \"kubernetes.io/projected/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-kube-api-access-q47dt\") pod \"e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") " pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.542647 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.542555    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:16.668925 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.668849    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"]
Jul 29 21:21:16.671424 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:21:16.671394    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd4f6f7_13cf_42c2_b238_7eed76a26b2f.slice/crio-f1d325e33ae23b3670cd9ad92b53b9d076d69b337b025865b3cff60fc93ee659 WatchSource:0}: Error finding container f1d325e33ae23b3670cd9ad92b53b9d076d69b337b025865b3cff60fc93ee659: Status 404 returned error can't find the container with id f1d325e33ae23b3670cd9ad92b53b9d076d69b337b025865b3cff60fc93ee659
Jul 29 21:21:16.676340 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.676301    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:21:16.676424 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.676388    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:21:16.676474 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:16.676428    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:21:17.664025 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:17.663987    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" event={"ID":"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f","Type":"ContainerStarted","Data":"2163a49451b0afad53af13864650db0112d3025c7ceb4b093f044d777f5d2366"}
Jul 29 21:21:17.664427 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:17.664039    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" event={"ID":"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f","Type":"ContainerStarted","Data":"f1d325e33ae23b3670cd9ad92b53b9d076d69b337b025865b3cff60fc93ee659"}
Jul 29 21:21:17.682918 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:17.682874    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" podStartSLOduration=1.6828604010000001 podStartE2EDuration="1.682860401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:21:17.681242883 +0000 UTC m=+2093.277534584" watchObservedRunningTime="2026-07-29 21:21:17.682860401 +0000 UTC m=+2093.279152085"
Jul 29 21:21:18.542818 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:18.542782    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:18.547459 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:18.547434    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:18.668904 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:18.668878    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:18.669807 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:18.669790    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:34.711349 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:34.711308    2584 patch_prober.go:28] interesting pod/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 29 21:21:34.711768 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:34.711394    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 21:21:35.748286 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:35.748196    2584 generic.go:356] "Generic (PLEG): container finished" podID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerID="2163a49451b0afad53af13864650db0112d3025c7ceb4b093f044d777f5d2366" exitCode=0
Jul 29 21:21:35.748286 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:35.748268    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" event={"ID":"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f","Type":"ContainerDied","Data":"2163a49451b0afad53af13864650db0112d3025c7ceb4b093f044d777f5d2366"}
Jul 29 21:21:35.748707 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:35.748620    2584 scope.go:117] "RemoveContainer" containerID="2163a49451b0afad53af13864650db0112d3025c7ceb4b093f044d777f5d2366"
Jul 29 21:21:35.749624 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:35.749599    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:21:35.749694 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:35.749679    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:21:35.749779 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:35.749718    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:21:36.758119 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:36.758066    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" event={"ID":"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f","Type":"ContainerStarted","Data":"7f3617c9da1a3871be3bf6c0bc94dd8cd1efb5887e1c705425c247e733d4e929"}
Jul 29 21:21:37.543642 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:37.543605    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:37.543863 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:37.543841    2584 patch_prober.go:28] interesting pod/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 29 21:21:37.543915 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:37.543889    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 29 21:21:38.543883 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:38.543848    2584 patch_prober.go:28] interesting pod/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 29 21:21:38.544306 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:38.543905    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 29 21:21:39.542850 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:39.542819    2584 patch_prober.go:28] interesting pod/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 29 21:21:39.543031 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:39.542865    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 29 21:21:40.543446 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:40.543413    2584 patch_prober.go:28] interesting pod/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 29 21:21:40.543891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:40.543470    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 29 21:21:41.546528 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:41.546495    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:41.546910 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:41.546859    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:21:41.547495 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:21:41.547475    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:23:24.979403 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:23:24.979372    2584 scope.go:117] "RemoveContainer" containerID="5e0fc1fd745fec4143604c2d0ae32bf9750096927d281765170a5cb883dc4251"
Jul 29 21:24:24.997924 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:24:24.997894    2584 scope.go:117] "RemoveContainer" containerID="33ccc922060b92b8c40d3d3545c383fff7a2be8cbf38c610b89143bbe0f56e14"
Jul 29 21:24:25.005720 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:24:25.005697    2584 scope.go:117] "RemoveContainer" containerID="d236fd3a7948049f9727dc94d126402632f1c08884feb0890afa88eff9f78ec0"
Jul 29 21:25:00.808061 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:00.808018    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn"]
Jul 29 21:25:00.811386 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:00.811365    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn"
Jul 29 21:25:00.814738 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:00.814717    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:25:00.814862 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:00.814719    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\""
Jul 29 21:25:00.852412 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:00.852379    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn"]
Jul 29 21:25:00.924365 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:00.924326    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d74dc1-3896-48e4-9724-b3556b1693f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn\" (UID: \"88d74dc1-3896-48e4-9724-b3556b1693f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn"
Jul 29 21:25:01.025403 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:01.025369    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d74dc1-3896-48e4-9724-b3556b1693f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn\" (UID: \"88d74dc1-3896-48e4-9724-b3556b1693f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn"
Jul 29 21:25:01.025979 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:01.025959    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d74dc1-3896-48e4-9724-b3556b1693f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn\" (UID: \"88d74dc1-3896-48e4-9724-b3556b1693f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn"
Jul 29 21:25:01.121326 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:01.121293    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn"
Jul 29 21:25:01.268606 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:01.268580    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn"]
Jul 29 21:25:01.270429 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:25:01.270403    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d74dc1_3896_48e4_9724_b3556b1693f5.slice/crio-13090accef9dd387e43c49cc17f724d915bda8081beee4f7dd359ccb9424a53a WatchSource:0}: Error finding container 13090accef9dd387e43c49cc17f724d915bda8081beee4f7dd359ccb9424a53a: Status 404 returned error can't find the container with id 13090accef9dd387e43c49cc17f724d915bda8081beee4f7dd359ccb9424a53a
Jul 29 21:25:01.682537 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:01.682502    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn" event={"ID":"88d74dc1-3896-48e4-9724-b3556b1693f5","Type":"ContainerStarted","Data":"cfb6f049ec011542ebe24883b44b62af3b270421a9b0371db515bb6dbfefa145"}
Jul 29 21:25:01.682716 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:01.682558    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn" event={"ID":"88d74dc1-3896-48e4-9724-b3556b1693f5","Type":"ContainerStarted","Data":"13090accef9dd387e43c49cc17f724d915bda8081beee4f7dd359ccb9424a53a"}
Jul 29 21:25:01.706017 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:01.705964    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn" podStartSLOduration=1.705946287 podStartE2EDuration="1.705946287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:25:01.703666525 +0000 UTC m=+2317.299958225" watchObservedRunningTime="2026-07-29 21:25:01.705946287 +0000 UTC m=+2317.302237971"
Jul 29 21:25:02.688402 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:02.688361    2584 generic.go:356] "Generic (PLEG): container finished" podID="88d74dc1-3896-48e4-9724-b3556b1693f5" containerID="cfb6f049ec011542ebe24883b44b62af3b270421a9b0371db515bb6dbfefa145" exitCode=0
Jul 29 21:25:02.688793 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:02.688409    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn" event={"ID":"88d74dc1-3896-48e4-9724-b3556b1693f5","Type":"ContainerDied","Data":"cfb6f049ec011542ebe24883b44b62af3b270421a9b0371db515bb6dbfefa145"}
Jul 29 21:25:03.824453 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:03.824429    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn"
Jul 29 21:25:03.952668 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:03.952573    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d74dc1-3896-48e4-9724-b3556b1693f5-maas-api-service-ca\") pod \"88d74dc1-3896-48e4-9724-b3556b1693f5\" (UID: \"88d74dc1-3896-48e4-9724-b3556b1693f5\") "
Jul 29 21:25:03.952950 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:03.952926    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d74dc1-3896-48e4-9724-b3556b1693f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88d74dc1-3896-48e4-9724-b3556b1693f5" (UID: "88d74dc1-3896-48e4-9724-b3556b1693f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:25:04.053460 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:04.053411    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d74dc1-3896-48e4-9724-b3556b1693f5-maas-api-service-ca\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:25:04.698191 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:04.698166    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn"
Jul 29 21:25:04.698191 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:04.698184    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn" event={"ID":"88d74dc1-3896-48e4-9724-b3556b1693f5","Type":"ContainerDied","Data":"13090accef9dd387e43c49cc17f724d915bda8081beee4f7dd359ccb9424a53a"}
Jul 29 21:25:04.698425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:04.698238    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13090accef9dd387e43c49cc17f724d915bda8081beee4f7dd359ccb9424a53a"
Jul 29 21:25:25.026069 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:25:25.025978    2584 scope.go:117] "RemoveContainer" containerID="644b66cc201f7dbd0c15d7baf5669c18801f1e2968cf6a2f56c7335d29a97c0f"
Jul 29 21:30:00.142659 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.142577    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756010-kbj6g"]
Jul 29 21:30:00.143126 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.142898    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d74dc1-3896-48e4-9724-b3556b1693f5" containerName="revoke-keys"
Jul 29 21:30:00.143126 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.142909    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d74dc1-3896-48e4-9724-b3556b1693f5" containerName="revoke-keys"
Jul 29 21:30:00.143126 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.142974    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d74dc1-3896-48e4-9724-b3556b1693f5" containerName="revoke-keys"
Jul 29 21:30:00.145818 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.145796    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq"]
Jul 29 21:30:00.145955 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.145924    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-kbj6g"
Jul 29 21:30:00.148056 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.148038    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\""
Jul 29 21:30:00.149484 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.149466    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq"
Jul 29 21:30:00.156586 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.156569    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-kbj6g"
Jul 29 21:30:00.163704 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.163684    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq"
Jul 29 21:30:00.170987 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.170958    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756010-kbj6g"]
Jul 29 21:30:00.176980 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.176904    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq"]
Jul 29 21:30:00.330283 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.330257    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq"]
Jul 29 21:30:00.332801 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:30:00.332751    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f7c21f_5063_4f9d_9cca_81cf7b567783.slice/crio-ef40a2f7e97c9aa11f1fffe1578449f02f22a661902f2fb6fb67627e7c8f21c3 WatchSource:0}: Error finding container ef40a2f7e97c9aa11f1fffe1578449f02f22a661902f2fb6fb67627e7c8f21c3: Status 404 returned error can't find the container with id ef40a2f7e97c9aa11f1fffe1578449f02f22a661902f2fb6fb67627e7c8f21c3
Jul 29 21:30:00.508974 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.508952    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756010-kbj6g"]
Jul 29 21:30:00.511393 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:30:00.511369    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24497491_89d5_4128_8ac7_c50e65201d08.slice/crio-ab0b3982a43ce72a29c35486ceb89467219cab1dfe69013ec471a69b451b1404 WatchSource:0}: Error finding container ab0b3982a43ce72a29c35486ceb89467219cab1dfe69013ec471a69b451b1404: Status 404 returned error can't find the container with id ab0b3982a43ce72a29c35486ceb89467219cab1dfe69013ec471a69b451b1404
Jul 29 21:30:00.994904 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.994869    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-kbj6g" event={"ID":"24497491-89d5-4128-8ac7-c50e65201d08","Type":"ContainerStarted","Data":"f5826e9676835d998a24dffba28e69eb2e7886d4f5fd8c44382bfddd7561ebf7"}
Jul 29 21:30:00.995099 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.994928    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-kbj6g" event={"ID":"24497491-89d5-4128-8ac7-c50e65201d08","Type":"ContainerStarted","Data":"ab0b3982a43ce72a29c35486ceb89467219cab1dfe69013ec471a69b451b1404"}
Jul 29 21:30:00.996355 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.996327    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq" event={"ID":"f8f7c21f-5063-4f9d-9cca-81cf7b567783","Type":"ContainerStarted","Data":"606b30cf4a3ab0c453856f639fe2bd70432ab452b4a677a3f33185b7fa94d0fa"}
Jul 29 21:30:00.996481 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:00.996377    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq" event={"ID":"f8f7c21f-5063-4f9d-9cca-81cf7b567783","Type":"ContainerStarted","Data":"ef40a2f7e97c9aa11f1fffe1578449f02f22a661902f2fb6fb67627e7c8f21c3"}
Jul 29 21:30:01.009795 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:01.009749    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-kbj6g" podStartSLOduration=1.009732655 podStartE2EDuration="1.009732655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:30:01.008344279 +0000 UTC m=+2616.604635964" watchObservedRunningTime="2026-07-29 21:30:01.009732655 +0000 UTC m=+2616.606024340"
Jul 29 21:30:01.022587 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:01.022525    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq" podStartSLOduration=1.022510279 podStartE2EDuration="1.022510279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:30:01.022108201 +0000 UTC m=+2616.618399884" watchObservedRunningTime="2026-07-29 21:30:01.022510279 +0000 UTC m=+2616.618801964"
Jul 29 21:30:01.401828 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:01.401805    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"]
Jul 29 21:30:01.402175 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:01.402005    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq" podUID="9b20990f-2284-4c68-a65f-add383f091e3" containerName="istio-proxy" containerID="cri-o://eb5284fe699ea0399d36620b062116ec90a17de7d783b31f9cb2d8e620961cc0" gracePeriod=30
Jul 29 21:30:01.766764 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:01.764236    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9"]
Jul 29 21:30:01.769823 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:01.768929    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9"
Jul 29 21:30:01.772003 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:01.771978    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:30:01.773578 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:01.773552    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9"]
Jul 29 21:30:01.878690 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:01.878657    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14369530-7b13-4e47-b76a-522764804b34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-58463a07-tlvh9\" (UID: \"14369530-7b13-4e47-b76a-522764804b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9"
Jul 29 21:30:01.979760 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:01.979721    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14369530-7b13-4e47-b76a-522764804b34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-58463a07-tlvh9\" (UID: \"14369530-7b13-4e47-b76a-522764804b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9"
Jul 29 21:30:01.980401 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:01.980381    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14369530-7b13-4e47-b76a-522764804b34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-58463a07-tlvh9\" (UID: \"14369530-7b13-4e47-b76a-522764804b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9"
Jul 29 21:30:02.002325 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:02.002281    2584 generic.go:356] "Generic (PLEG): container finished" podID="24497491-89d5-4128-8ac7-c50e65201d08" containerID="f5826e9676835d998a24dffba28e69eb2e7886d4f5fd8c44382bfddd7561ebf7" exitCode=0
Jul 29 21:30:02.002503 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:02.002370    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-kbj6g" event={"ID":"24497491-89d5-4128-8ac7-c50e65201d08","Type":"ContainerDied","Data":"f5826e9676835d998a24dffba28e69eb2e7886d4f5fd8c44382bfddd7561ebf7"}
Jul 29 21:30:02.003836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:02.003813    2584 generic.go:356] "Generic (PLEG): container finished" podID="f8f7c21f-5063-4f9d-9cca-81cf7b567783" containerID="606b30cf4a3ab0c453856f639fe2bd70432ab452b4a677a3f33185b7fa94d0fa" exitCode=0
Jul 29 21:30:02.003909 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:02.003861    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq" event={"ID":"f8f7c21f-5063-4f9d-9cca-81cf7b567783","Type":"ContainerDied","Data":"606b30cf4a3ab0c453856f639fe2bd70432ab452b4a677a3f33185b7fa94d0fa"}
Jul 29 21:30:02.081119 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:02.081030    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9"
Jul 29 21:30:02.217564 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:02.217537    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9"]
Jul 29 21:30:02.219951 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:30:02.219918    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14369530_7b13_4e47_b76a_522764804b34.slice/crio-7a010aa40e28d63183fa551624f3944d808913e67a91a93edf4096ec6a1a1998 WatchSource:0}: Error finding container 7a010aa40e28d63183fa551624f3944d808913e67a91a93edf4096ec6a1a1998: Status 404 returned error can't find the container with id 7a010aa40e28d63183fa551624f3944d808913e67a91a93edf4096ec6a1a1998
Jul 29 21:30:03.010057 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:03.010022    2584 generic.go:356] "Generic (PLEG): container finished" podID="14369530-7b13-4e47-b76a-522764804b34" containerID="62e6da4790245534ee1cc41bef80341c9c43ac5bf77e8f4db37f081529f9ae88" exitCode=0
Jul 29 21:30:03.010459 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:03.010108    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9" event={"ID":"14369530-7b13-4e47-b76a-522764804b34","Type":"ContainerDied","Data":"62e6da4790245534ee1cc41bef80341c9c43ac5bf77e8f4db37f081529f9ae88"}
Jul 29 21:30:03.010459 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:03.010185    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9" event={"ID":"14369530-7b13-4e47-b76a-522764804b34","Type":"ContainerStarted","Data":"7a010aa40e28d63183fa551624f3944d808913e67a91a93edf4096ec6a1a1998"}
Jul 29 21:30:03.151426 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:03.151403    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-kbj6g"
Jul 29 21:30:03.168031 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:03.167991    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq"
Jul 29 21:30:04.015911 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:04.015878    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq"
Jul 29 21:30:04.016368 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:04.015910    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq" event={"ID":"f8f7c21f-5063-4f9d-9cca-81cf7b567783","Type":"ContainerDied","Data":"ef40a2f7e97c9aa11f1fffe1578449f02f22a661902f2fb6fb67627e7c8f21c3"}
Jul 29 21:30:04.016368 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:04.015961    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef40a2f7e97c9aa11f1fffe1578449f02f22a661902f2fb6fb67627e7c8f21c3"
Jul 29 21:30:04.017140 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:04.017122    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-kbj6g"
Jul 29 21:30:04.017235 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:04.017170    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-kbj6g" event={"ID":"24497491-89d5-4128-8ac7-c50e65201d08","Type":"ContainerDied","Data":"ab0b3982a43ce72a29c35486ceb89467219cab1dfe69013ec471a69b451b1404"}
Jul 29 21:30:04.017235 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:04.017203    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab0b3982a43ce72a29c35486ceb89467219cab1dfe69013ec471a69b451b1404"
Jul 29 21:30:04.036377 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:04.036346    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn"]
Jul 29 21:30:04.040035 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:04.040011    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c67e2c-57vqn"]
Jul 29 21:30:04.150960 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:04.150934    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9"
Jul 29 21:30:04.197519 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:04.197486    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14369530-7b13-4e47-b76a-522764804b34-maas-api-service-ca\") pod \"14369530-7b13-4e47-b76a-522764804b34\" (UID: \"14369530-7b13-4e47-b76a-522764804b34\") "
Jul 29 21:30:04.197885 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:04.197866    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14369530-7b13-4e47-b76a-522764804b34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14369530-7b13-4e47-b76a-522764804b34" (UID: "14369530-7b13-4e47-b76a-522764804b34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:30:04.298312 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:04.298227    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14369530-7b13-4e47-b76a-522764804b34-maas-api-service-ca\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:04.866203 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:04.866169    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d74dc1-3896-48e4-9724-b3556b1693f5" path="/var/lib/kubelet/pods/88d74dc1-3896-48e4-9724-b3556b1693f5/volumes"
Jul 29 21:30:05.022672 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:05.022639    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9" event={"ID":"14369530-7b13-4e47-b76a-522764804b34","Type":"ContainerDied","Data":"7a010aa40e28d63183fa551624f3944d808913e67a91a93edf4096ec6a1a1998"}
Jul 29 21:30:05.023096 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:05.022681    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9"
Jul 29 21:30:05.023096 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:05.022685    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a010aa40e28d63183fa551624f3944d808913e67a91a93edf4096ec6a1a1998"
Jul 29 21:30:06.654888 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.654862    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:30:06.719063 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.718975    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhv66\" (UniqueName: \"kubernetes.io/projected/9b20990f-2284-4c68-a65f-add383f091e3-kube-api-access-vhv66\") pod \"9b20990f-2284-4c68-a65f-add383f091e3\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") "
Jul 29 21:30:06.719063 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.719018    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-istio-data\") pod \"9b20990f-2284-4c68-a65f-add383f091e3\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") "
Jul 29 21:30:06.719063 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.719057    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-workload-certs\") pod \"9b20990f-2284-4c68-a65f-add383f091e3\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") "
Jul 29 21:30:06.719348 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.719106    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b20990f-2284-4c68-a65f-add383f091e3-istiod-ca-cert\") pod \"9b20990f-2284-4c68-a65f-add383f091e3\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") "
Jul 29 21:30:06.719348 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.719168    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-credential-socket\") pod \"9b20990f-2284-4c68-a65f-add383f091e3\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") "
Jul 29 21:30:06.719348 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.719199    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b20990f-2284-4c68-a65f-add383f091e3-istio-token\") pod \"9b20990f-2284-4c68-a65f-add383f091e3\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") "
Jul 29 21:30:06.719348 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.719228    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-workload-socket\") pod \"9b20990f-2284-4c68-a65f-add383f091e3\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") "
Jul 29 21:30:06.719348 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.719277    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b20990f-2284-4c68-a65f-add383f091e3-istio-podinfo\") pod \"9b20990f-2284-4c68-a65f-add383f091e3\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") "
Jul 29 21:30:06.719348 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.719315    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-istio-envoy\") pod \"9b20990f-2284-4c68-a65f-add383f091e3\" (UID: \"9b20990f-2284-4c68-a65f-add383f091e3\") "
Jul 29 21:30:06.719649 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.719406    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b20990f-2284-4c68-a65f-add383f091e3" (UID: "9b20990f-2284-4c68-a65f-add383f091e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:30:06.719649 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.719560    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b20990f-2284-4c68-a65f-add383f091e3" (UID: "9b20990f-2284-4c68-a65f-add383f091e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:30:06.719649 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.719572    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b20990f-2284-4c68-a65f-add383f091e3" (UID: "9b20990f-2284-4c68-a65f-add383f091e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:30:06.719649 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.719614    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:06.719866 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.719645    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b20990f-2284-4c68-a65f-add383f091e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b20990f-2284-4c68-a65f-add383f091e3" (UID: "9b20990f-2284-4c68-a65f-add383f091e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:30:06.720231 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.720208    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b20990f-2284-4c68-a65f-add383f091e3" (UID: "9b20990f-2284-4c68-a65f-add383f091e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:30:06.721382 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.721359    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b20990f-2284-4c68-a65f-add383f091e3-kube-api-access-vhv66" (OuterVolumeSpecName: "kube-api-access-vhv66") pod "9b20990f-2284-4c68-a65f-add383f091e3" (UID: "9b20990f-2284-4c68-a65f-add383f091e3"). InnerVolumeSpecName "kube-api-access-vhv66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:30:06.721522 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.721507    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b20990f-2284-4c68-a65f-add383f091e3" (UID: "9b20990f-2284-4c68-a65f-add383f091e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:30:06.721812 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.721786    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b20990f-2284-4c68-a65f-add383f091e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b20990f-2284-4c68-a65f-add383f091e3" (UID: "9b20990f-2284-4c68-a65f-add383f091e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:30:06.721812 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.721790    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b20990f-2284-4c68-a65f-add383f091e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b20990f-2284-4c68-a65f-add383f091e3" (UID: "9b20990f-2284-4c68-a65f-add383f091e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:30:06.820364 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.820322    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b20990f-2284-4c68-a65f-add383f091e3-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:06.820364 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.820358    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:06.820364 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.820369    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhv66\" (UniqueName: \"kubernetes.io/projected/9b20990f-2284-4c68-a65f-add383f091e3-kube-api-access-vhv66\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:06.820601 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.820380    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:06.820601 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.820388    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b20990f-2284-4c68-a65f-add383f091e3-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:06.820601 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.820396    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:06.820601 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.820404    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b20990f-2284-4c68-a65f-add383f091e3-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:06.820601 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:06.820412    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b20990f-2284-4c68-a65f-add383f091e3-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:07.033902 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:07.033813    2584 generic.go:356] "Generic (PLEG): container finished" podID="9b20990f-2284-4c68-a65f-add383f091e3" containerID="eb5284fe699ea0399d36620b062116ec90a17de7d783b31f9cb2d8e620961cc0" exitCode=0
Jul 29 21:30:07.033902 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:07.033859    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"
Jul 29 21:30:07.034089 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:07.033895    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq" event={"ID":"9b20990f-2284-4c68-a65f-add383f091e3","Type":"ContainerDied","Data":"eb5284fe699ea0399d36620b062116ec90a17de7d783b31f9cb2d8e620961cc0"}
Jul 29 21:30:07.034089 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:07.033945    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq" event={"ID":"9b20990f-2284-4c68-a65f-add383f091e3","Type":"ContainerDied","Data":"c640f3ea2b98b4e33f70166d7b55d0ee53c29d893ef23c1674cdd9243a616504"}
Jul 29 21:30:07.034089 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:07.033966    2584 scope.go:117] "RemoveContainer" containerID="eb5284fe699ea0399d36620b062116ec90a17de7d783b31f9cb2d8e620961cc0"
Jul 29 21:30:07.043241 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:07.043217    2584 scope.go:117] "RemoveContainer" containerID="f087496413cb212f0d4b9131b0e5a602e76ee644387681777456f1bc986f4f5a"
Jul 29 21:30:07.050080 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:07.050055    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"]
Jul 29 21:30:07.054380 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:07.054356    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7c67e2c-openshift-default-5645b6b5dd-zsfsq"]
Jul 29 21:30:07.055070 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:07.055055    2584 scope.go:117] "RemoveContainer" containerID="eb5284fe699ea0399d36620b062116ec90a17de7d783b31f9cb2d8e620961cc0"
Jul 29 21:30:07.055396 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:30:07.055377    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5284fe699ea0399d36620b062116ec90a17de7d783b31f9cb2d8e620961cc0\": container with ID starting with eb5284fe699ea0399d36620b062116ec90a17de7d783b31f9cb2d8e620961cc0 not found: ID does not exist" containerID="eb5284fe699ea0399d36620b062116ec90a17de7d783b31f9cb2d8e620961cc0"
Jul 29 21:30:07.055481 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:07.055407    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5284fe699ea0399d36620b062116ec90a17de7d783b31f9cb2d8e620961cc0"} err="failed to get container status \"eb5284fe699ea0399d36620b062116ec90a17de7d783b31f9cb2d8e620961cc0\": rpc error: code = NotFound desc = could not find container \"eb5284fe699ea0399d36620b062116ec90a17de7d783b31f9cb2d8e620961cc0\": container with ID starting with eb5284fe699ea0399d36620b062116ec90a17de7d783b31f9cb2d8e620961cc0 not found: ID does not exist"
Jul 29 21:30:07.055481 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:07.055430    2584 scope.go:117] "RemoveContainer" containerID="f087496413cb212f0d4b9131b0e5a602e76ee644387681777456f1bc986f4f5a"
Jul 29 21:30:07.055671 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:30:07.055653    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f087496413cb212f0d4b9131b0e5a602e76ee644387681777456f1bc986f4f5a\": container with ID starting with f087496413cb212f0d4b9131b0e5a602e76ee644387681777456f1bc986f4f5a not found: ID does not exist" containerID="f087496413cb212f0d4b9131b0e5a602e76ee644387681777456f1bc986f4f5a"
Jul 29 21:30:07.055722 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:07.055678    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f087496413cb212f0d4b9131b0e5a602e76ee644387681777456f1bc986f4f5a"} err="failed to get container status \"f087496413cb212f0d4b9131b0e5a602e76ee644387681777456f1bc986f4f5a\": rpc error: code = NotFound desc = could not find container \"f087496413cb212f0d4b9131b0e5a602e76ee644387681777456f1bc986f4f5a\": container with ID starting with f087496413cb212f0d4b9131b0e5a602e76ee644387681777456f1bc986f4f5a not found: ID does not exist"
Jul 29 21:30:08.865590 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:08.865555    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b20990f-2284-4c68-a65f-add383f091e3" path="/var/lib/kubelet/pods/9b20990f-2284-4c68-a65f-add383f091e3/volumes"
Jul 29 21:30:17.294535 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:17.294497    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq"]
Jul 29 21:30:17.298773 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:17.298573    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-58463a07-29756010-8shsq"]
Jul 29 21:30:17.521295 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:17.521260    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9"]
Jul 29 21:30:17.525497 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:17.525471    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58463a07-tlvh9"]
Jul 29 21:30:18.865036 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:18.865000    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14369530-7b13-4e47-b76a-522764804b34" path="/var/lib/kubelet/pods/14369530-7b13-4e47-b76a-522764804b34/volumes"
Jul 29 21:30:18.865425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:18.865290    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8f7c21f-5063-4f9d-9cca-81cf7b567783" path="/var/lib/kubelet/pods/f8f7c21f-5063-4f9d-9cca-81cf7b567783/volumes"
Jul 29 21:30:24.078822 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.078790    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"]
Jul 29 21:30:24.079224 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.078996    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerName="istio-proxy" containerID="cri-o://7f3617c9da1a3871be3bf6c0bc94dd8cd1efb5887e1c705425c247e733d4e929" gracePeriod=30
Jul 29 21:30:24.962713 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.962680    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"]
Jul 29 21:30:24.963002 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.962990    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b20990f-2284-4c68-a65f-add383f091e3" containerName="istio-proxy"
Jul 29 21:30:24.963049 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963003    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b20990f-2284-4c68-a65f-add383f091e3" containerName="istio-proxy"
Jul 29 21:30:24.963049 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963016    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f7c21f-5063-4f9d-9cca-81cf7b567783" containerName="cleanup"
Jul 29 21:30:24.963049 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963021    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f7c21f-5063-4f9d-9cca-81cf7b567783" containerName="cleanup"
Jul 29 21:30:24.963049 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963027    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24497491-89d5-4128-8ac7-c50e65201d08" containerName="cleanup"
Jul 29 21:30:24.963049 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963034    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="24497491-89d5-4128-8ac7-c50e65201d08" containerName="cleanup"
Jul 29 21:30:24.963049 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963044    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14369530-7b13-4e47-b76a-522764804b34" containerName="revoke-keys"
Jul 29 21:30:24.963049 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963049    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="14369530-7b13-4e47-b76a-522764804b34" containerName="revoke-keys"
Jul 29 21:30:24.963270 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963058    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b20990f-2284-4c68-a65f-add383f091e3" containerName="istio-proxy"
Jul 29 21:30:24.963270 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963063    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b20990f-2284-4c68-a65f-add383f091e3" containerName="istio-proxy"
Jul 29 21:30:24.963270 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963116    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="24497491-89d5-4128-8ac7-c50e65201d08" containerName="cleanup"
Jul 29 21:30:24.963270 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963125    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="14369530-7b13-4e47-b76a-522764804b34" containerName="revoke-keys"
Jul 29 21:30:24.963270 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963132    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b20990f-2284-4c68-a65f-add383f091e3" containerName="istio-proxy"
Jul 29 21:30:24.963270 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963137    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f7c21f-5063-4f9d-9cca-81cf7b567783" containerName="cleanup"
Jul 29 21:30:24.963270 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.963158    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b20990f-2284-4c68-a65f-add383f091e3" containerName="istio-proxy"
Jul 29 21:30:24.967330 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.967310    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:24.969700 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.969679    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a9af2a72-openshift-default-dockercfg-6w86b\""
Jul 29 21:30:24.978376 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:24.978352    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"]
Jul 29 21:30:25.081977 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.081951    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-workload-certs\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.082345 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.081988    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105002ba-65d9-40e5-9c3e-91e7f51ae790-istiod-ca-cert\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.082345 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.082006    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-credential-socket\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.082345 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.082056    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-workload-socket\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.082345 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.082077    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l548q\" (UniqueName: \"kubernetes.io/projected/105002ba-65d9-40e5-9c3e-91e7f51ae790-kube-api-access-l548q\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.082345 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.082132    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-data\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.082345 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.082198    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-envoy\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.082345 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.082245    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-podinfo\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.082345 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.082262    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-token\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.182852 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.182812    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-workload-socket\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.182852 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.182860    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l548q\" (UniqueName: \"kubernetes.io/projected/105002ba-65d9-40e5-9c3e-91e7f51ae790-kube-api-access-l548q\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.183097 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.182898    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-data\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.183097 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.182940    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-envoy\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.183097 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.182986    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-podinfo\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.183097 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.183010    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-token\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.183097 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.183038    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-workload-certs\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.183097 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.183074    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105002ba-65d9-40e5-9c3e-91e7f51ae790-istiod-ca-cert\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.183447 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.183100    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-credential-socket\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.183447 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.183271    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-workload-socket\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.183447 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.183393    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-workload-certs\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.183586 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.183508    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-credential-socket\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.183631 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.183579    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-data\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.183827 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.183806    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105002ba-65d9-40e5-9c3e-91e7f51ae790-istiod-ca-cert\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.185309 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.185288    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-envoy\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.185508 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.185490    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-podinfo\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.192041 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.191978    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l548q\" (UniqueName: \"kubernetes.io/projected/105002ba-65d9-40e5-9c3e-91e7f51ae790-kube-api-access-l548q\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.192604 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.192543    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-token\") pod \"e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") " pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.276954 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.276872    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:25.412402 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.412373    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"]
Jul 29 21:30:25.414938 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:30:25.414910    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105002ba_65d9_40e5_9c3e_91e7f51ae790.slice/crio-85e748a64a9ab262f3dea7822365f217ad309c992d9e22f89476193aeb570485 WatchSource:0}: Error finding container 85e748a64a9ab262f3dea7822365f217ad309c992d9e22f89476193aeb570485: Status 404 returned error can't find the container with id 85e748a64a9ab262f3dea7822365f217ad309c992d9e22f89476193aeb570485
Jul 29 21:30:25.417190 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.417159    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:30:25.417297 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.417239    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:30:25.417297 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:25.417283    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:30:26.120843 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:26.120805    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6" event={"ID":"105002ba-65d9-40e5-9c3e-91e7f51ae790","Type":"ContainerStarted","Data":"bfe6df1a8f6e585be6d4b753c091a721e0c67484c8b2f0f23c1378937fde68ed"}
Jul 29 21:30:26.121269 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:26.120865    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6" event={"ID":"105002ba-65d9-40e5-9c3e-91e7f51ae790","Type":"ContainerStarted","Data":"85e748a64a9ab262f3dea7822365f217ad309c992d9e22f89476193aeb570485"}
Jul 29 21:30:26.143867 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:26.143804    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6" podStartSLOduration=2.143784018 podStartE2EDuration="2.143784018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:30:26.143119748 +0000 UTC m=+2641.739411431" watchObservedRunningTime="2026-07-29 21:30:26.143784018 +0000 UTC m=+2641.740075701"
Jul 29 21:30:26.277823 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:26.277785    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:26.282529 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:26.282502    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:26.549536 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:26.549460    2584 patch_prober.go:28] interesting pod/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 21:30:26.549536 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:26.549512    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:30:27.126216 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:27.126183    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:27.127247 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:27.127230    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:29.347475 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.347447    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:30:29.522532 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.522493    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-workload-certs\") pod \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") "
Jul 29 21:30:29.522532 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.522531    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-credential-socket\") pod \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") "
Jul 29 21:30:29.522744 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.522564    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istiod-ca-cert\") pod \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") "
Jul 29 21:30:29.522744 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.522604    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-workload-socket\") pod \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") "
Jul 29 21:30:29.522744 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.522625    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-data\") pod \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") "
Jul 29 21:30:29.522744 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.522654    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-podinfo\") pod \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") "
Jul 29 21:30:29.522744 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.522677    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-token\") pod \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") "
Jul 29 21:30:29.523075 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.522746    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q47dt\" (UniqueName: \"kubernetes.io/projected/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-kube-api-access-q47dt\") pod \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") "
Jul 29 21:30:29.523075 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.522779    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-envoy\") pod \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\" (UID: \"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f\") "
Jul 29 21:30:29.523075 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.522812    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" (UID: "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:30:29.523075 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.522832    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" (UID: "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:30:29.523075 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.522978    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" (UID: "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:30:29.523075 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.523048    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" (UID: "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:30:29.523427 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.523091    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:29.523427 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.523110    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:29.523427 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.523133    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:29.523581 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.523519    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" (UID: "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:30:29.525243 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.525190    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-kube-api-access-q47dt" (OuterVolumeSpecName: "kube-api-access-q47dt") pod "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" (UID: "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f"). InnerVolumeSpecName "kube-api-access-q47dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:30:29.525243 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.525193    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" (UID: "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:30:29.525423 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.525367    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" (UID: "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:30:29.525486 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.525466    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" (UID: "bbd4f6f7-13cf-42c2-b238-7eed76a26b2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:30:29.623922 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.623843    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:29.623922 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.623871    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:29.623922 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.623880    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:29.623922 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.623888    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:29.623922 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.623896    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q47dt\" (UniqueName: \"kubernetes.io/projected/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-kube-api-access-q47dt\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:29.623922 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:29.623906    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:30.141336 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:30.141306    2584 generic.go:356] "Generic (PLEG): container finished" podID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerID="7f3617c9da1a3871be3bf6c0bc94dd8cd1efb5887e1c705425c247e733d4e929" exitCode=0
Jul 29 21:30:30.141511 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:30.141360    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"
Jul 29 21:30:30.141511 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:30.141381    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" event={"ID":"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f","Type":"ContainerDied","Data":"7f3617c9da1a3871be3bf6c0bc94dd8cd1efb5887e1c705425c247e733d4e929"}
Jul 29 21:30:30.141511 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:30.141424    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8" event={"ID":"bbd4f6f7-13cf-42c2-b238-7eed76a26b2f","Type":"ContainerDied","Data":"f1d325e33ae23b3670cd9ad92b53b9d076d69b337b025865b3cff60fc93ee659"}
Jul 29 21:30:30.141511 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:30.141446    2584 scope.go:117] "RemoveContainer" containerID="7f3617c9da1a3871be3bf6c0bc94dd8cd1efb5887e1c705425c247e733d4e929"
Jul 29 21:30:30.151418 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:30.151102    2584 scope.go:117] "RemoveContainer" containerID="2163a49451b0afad53af13864650db0112d3025c7ceb4b093f044d777f5d2366"
Jul 29 21:30:30.159280 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:30.159262    2584 scope.go:117] "RemoveContainer" containerID="7f3617c9da1a3871be3bf6c0bc94dd8cd1efb5887e1c705425c247e733d4e929"
Jul 29 21:30:30.159573 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:30:30.159552    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3617c9da1a3871be3bf6c0bc94dd8cd1efb5887e1c705425c247e733d4e929\": container with ID starting with 7f3617c9da1a3871be3bf6c0bc94dd8cd1efb5887e1c705425c247e733d4e929 not found: ID does not exist" containerID="7f3617c9da1a3871be3bf6c0bc94dd8cd1efb5887e1c705425c247e733d4e929"
Jul 29 21:30:30.159625 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:30.159583    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3617c9da1a3871be3bf6c0bc94dd8cd1efb5887e1c705425c247e733d4e929"} err="failed to get container status \"7f3617c9da1a3871be3bf6c0bc94dd8cd1efb5887e1c705425c247e733d4e929\": rpc error: code = NotFound desc = could not find container \"7f3617c9da1a3871be3bf6c0bc94dd8cd1efb5887e1c705425c247e733d4e929\": container with ID starting with 7f3617c9da1a3871be3bf6c0bc94dd8cd1efb5887e1c705425c247e733d4e929 not found: ID does not exist"
Jul 29 21:30:30.159625 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:30.159600    2584 scope.go:117] "RemoveContainer" containerID="2163a49451b0afad53af13864650db0112d3025c7ceb4b093f044d777f5d2366"
Jul 29 21:30:30.159843 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:30:30.159829    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2163a49451b0afad53af13864650db0112d3025c7ceb4b093f044d777f5d2366\": container with ID starting with 2163a49451b0afad53af13864650db0112d3025c7ceb4b093f044d777f5d2366 not found: ID does not exist" containerID="2163a49451b0afad53af13864650db0112d3025c7ceb4b093f044d777f5d2366"
Jul 29 21:30:30.159884 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:30.159849    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2163a49451b0afad53af13864650db0112d3025c7ceb4b093f044d777f5d2366"} err="failed to get container status \"2163a49451b0afad53af13864650db0112d3025c7ceb4b093f044d777f5d2366\": rpc error: code = NotFound desc = could not find container \"2163a49451b0afad53af13864650db0112d3025c7ceb4b093f044d777f5d2366\": container with ID starting with 2163a49451b0afad53af13864650db0112d3025c7ceb4b093f044d777f5d2366 not found: ID does not exist"
Jul 29 21:30:30.165171 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:30.165130    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"]
Jul 29 21:30:30.168672 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:30.168652    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-58463a07-openshift-default-7c456d659b-mnnj8"]
Jul 29 21:30:30.865042 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:30.865007    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" path="/var/lib/kubelet/pods/bbd4f6f7-13cf-42c2-b238-7eed76a26b2f/volumes"
Jul 29 21:30:41.196060 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:41.196027    2584 generic.go:356] "Generic (PLEG): container finished" podID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerID="bfe6df1a8f6e585be6d4b753c091a721e0c67484c8b2f0f23c1378937fde68ed" exitCode=0
Jul 29 21:30:41.196618 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:41.196107    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6" event={"ID":"105002ba-65d9-40e5-9c3e-91e7f51ae790","Type":"ContainerDied","Data":"bfe6df1a8f6e585be6d4b753c091a721e0c67484c8b2f0f23c1378937fde68ed"}
Jul 29 21:30:41.196687 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:41.196649    2584 scope.go:117] "RemoveContainer" containerID="bfe6df1a8f6e585be6d4b753c091a721e0c67484c8b2f0f23c1378937fde68ed"
Jul 29 21:30:41.197740 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:41.197713    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:30:41.197798 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:41.197780    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:30:41.197835 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:41.197812    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:30:42.126584 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:42.126541    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:42.206396 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:42.206350    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6" event={"ID":"105002ba-65d9-40e5-9c3e-91e7f51ae790","Type":"ContainerStarted","Data":"bd90e72e71c18e728200a70a14d4c699368a15575f8091b3959eae52b669e511"}
Jul 29 21:30:42.277957 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:42.277921    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:42.279015 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:42.278992    2584 patch_prober.go:28] interesting pod/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 29 21:30:42.279076 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:42.279040    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6" podUID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 29 21:30:43.277545 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:43.277507    2584 patch_prober.go:28] interesting pod/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 29 21:30:43.277950 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:43.277562    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6" podUID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 29 21:30:44.277258 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:44.277221    2584 patch_prober.go:28] interesting pod/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 29 21:30:44.277434 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:44.277279    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6" podUID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 29 21:30:45.278042 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.278008    2584 patch_prober.go:28] interesting pod/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 29 21:30:45.278466 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.278060    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6" podUID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 29 21:30:45.572557 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.572466    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w"]
Jul 29 21:30:45.572878 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.572860    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerName="istio-proxy"
Jul 29 21:30:45.572972 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.572880    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerName="istio-proxy"
Jul 29 21:30:45.573030 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.572972    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerName="istio-proxy"
Jul 29 21:30:45.574961 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.574940    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w"
Jul 29 21:30:45.576976 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.576950    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:30:45.576976 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.576969    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\""
Jul 29 21:30:45.583396 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.583364    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w"]
Jul 29 21:30:45.658270 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.658233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a27a492-1f97-4b2c-b76f-8fb9e27227fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w\" (UID: \"1a27a492-1f97-4b2c-b76f-8fb9e27227fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w"
Jul 29 21:30:45.759357 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.759320    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a27a492-1f97-4b2c-b76f-8fb9e27227fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w\" (UID: \"1a27a492-1f97-4b2c-b76f-8fb9e27227fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w"
Jul 29 21:30:45.759947 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.759929    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a27a492-1f97-4b2c-b76f-8fb9e27227fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w\" (UID: \"1a27a492-1f97-4b2c-b76f-8fb9e27227fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w"
Jul 29 21:30:45.886187 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:45.886127    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w"
Jul 29 21:30:46.055692 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:46.055656    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w"]
Jul 29 21:30:46.059903 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:30:46.059872    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a27a492_1f97_4b2c_b76f_8fb9e27227fe.slice/crio-49a78c740b6eefb2b55c506c4aba311d32062eac9975260d760df4b62ac17339 WatchSource:0}: Error finding container 49a78c740b6eefb2b55c506c4aba311d32062eac9975260d760df4b62ac17339: Status 404 returned error can't find the container with id 49a78c740b6eefb2b55c506c4aba311d32062eac9975260d760df4b62ac17339
Jul 29 21:30:46.229024 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:46.228935    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w" event={"ID":"1a27a492-1f97-4b2c-b76f-8fb9e27227fe","Type":"ContainerStarted","Data":"49a78c740b6eefb2b55c506c4aba311d32062eac9975260d760df4b62ac17339"}
Jul 29 21:30:46.281472 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:46.281439    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:46.281842 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:46.281703    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:46.282376 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:46.282359    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:30:47.236187 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:47.236130    2584 generic.go:356] "Generic (PLEG): container finished" podID="1a27a492-1f97-4b2c-b76f-8fb9e27227fe" containerID="a2597ba131880aecb80ff494972f63dc5e0fae806491ddebd610bc0190c0e98d" exitCode=0
Jul 29 21:30:47.236365 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:47.236185    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w" event={"ID":"1a27a492-1f97-4b2c-b76f-8fb9e27227fe","Type":"ContainerDied","Data":"a2597ba131880aecb80ff494972f63dc5e0fae806491ddebd610bc0190c0e98d"}
Jul 29 21:30:48.373226 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:48.373194    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w"
Jul 29 21:30:48.478917 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:48.478879    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a27a492-1f97-4b2c-b76f-8fb9e27227fe-maas-api-service-ca\") pod \"1a27a492-1f97-4b2c-b76f-8fb9e27227fe\" (UID: \"1a27a492-1f97-4b2c-b76f-8fb9e27227fe\") "
Jul 29 21:30:48.479260 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:48.479235    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a27a492-1f97-4b2c-b76f-8fb9e27227fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a27a492-1f97-4b2c-b76f-8fb9e27227fe" (UID: "1a27a492-1f97-4b2c-b76f-8fb9e27227fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:30:48.580293 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:48.580201    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a27a492-1f97-4b2c-b76f-8fb9e27227fe-maas-api-service-ca\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:30:49.246838 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:49.246760    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w"
Jul 29 21:30:49.246838 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:49.246793    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w" event={"ID":"1a27a492-1f97-4b2c-b76f-8fb9e27227fe","Type":"ContainerDied","Data":"49a78c740b6eefb2b55c506c4aba311d32062eac9975260d760df4b62ac17339"}
Jul 29 21:30:49.246838 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:49.246836    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a78c740b6eefb2b55c506c4aba311d32062eac9975260d760df4b62ac17339"
Jul 29 21:30:51.349099 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:51.349063    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w"]
Jul 29 21:30:51.352225 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:51.352201    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9af2a72-2qc9w"]
Jul 29 21:30:52.866058 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:52.866026    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a27a492-1f97-4b2c-b76f-8fb9e27227fe" path="/var/lib/kubelet/pods/1a27a492-1f97-4b2c-b76f-8fb9e27227fe/volumes"
Jul 29 21:30:57.736643 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:57.736608    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"]
Jul 29 21:30:57.737032 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:30:57.736816    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6" podUID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerName="istio-proxy" containerID="cri-o://bd90e72e71c18e728200a70a14d4c699368a15575f8091b3959eae52b669e511" gracePeriod=30
Jul 29 21:31:01.283956 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:01.283915    2584 patch_prober.go:28] interesting pod/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 21:31:01.284368 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:01.283976    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6" podUID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:31:02.986515 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:02.986492    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:31:03.108731 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.108698    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-workload-certs\") pod \"105002ba-65d9-40e5-9c3e-91e7f51ae790\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") "
Jul 29 21:31:03.108908 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.108743    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-data\") pod \"105002ba-65d9-40e5-9c3e-91e7f51ae790\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") "
Jul 29 21:31:03.108908 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.108777    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l548q\" (UniqueName: \"kubernetes.io/projected/105002ba-65d9-40e5-9c3e-91e7f51ae790-kube-api-access-l548q\") pod \"105002ba-65d9-40e5-9c3e-91e7f51ae790\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") "
Jul 29 21:31:03.108908 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.108812    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-workload-socket\") pod \"105002ba-65d9-40e5-9c3e-91e7f51ae790\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") "
Jul 29 21:31:03.108908 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.108828    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-token\") pod \"105002ba-65d9-40e5-9c3e-91e7f51ae790\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") "
Jul 29 21:31:03.108908 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.108873    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105002ba-65d9-40e5-9c3e-91e7f51ae790-istiod-ca-cert\") pod \"105002ba-65d9-40e5-9c3e-91e7f51ae790\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") "
Jul 29 21:31:03.108908 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.108889    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-credential-socket\") pod \"105002ba-65d9-40e5-9c3e-91e7f51ae790\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") "
Jul 29 21:31:03.109254 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.108949    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-podinfo\") pod \"105002ba-65d9-40e5-9c3e-91e7f51ae790\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") "
Jul 29 21:31:03.109254 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.108972    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-envoy\") pod \"105002ba-65d9-40e5-9c3e-91e7f51ae790\" (UID: \"105002ba-65d9-40e5-9c3e-91e7f51ae790\") "
Jul 29 21:31:03.109254 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.109087    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "105002ba-65d9-40e5-9c3e-91e7f51ae790" (UID: "105002ba-65d9-40e5-9c3e-91e7f51ae790"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:31:03.109416 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.109242    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "105002ba-65d9-40e5-9c3e-91e7f51ae790" (UID: "105002ba-65d9-40e5-9c3e-91e7f51ae790"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:31:03.109416 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.109276    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "105002ba-65d9-40e5-9c3e-91e7f51ae790" (UID: "105002ba-65d9-40e5-9c3e-91e7f51ae790"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:31:03.109416 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.109335    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:31:03.109416 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.109354    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:31:03.109416 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.109362    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105002ba-65d9-40e5-9c3e-91e7f51ae790-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "105002ba-65d9-40e5-9c3e-91e7f51ae790" (UID: "105002ba-65d9-40e5-9c3e-91e7f51ae790"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:31:03.109704 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.109683    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-data" (OuterVolumeSpecName: "istio-data") pod "105002ba-65d9-40e5-9c3e-91e7f51ae790" (UID: "105002ba-65d9-40e5-9c3e-91e7f51ae790"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:31:03.111084 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.111061    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105002ba-65d9-40e5-9c3e-91e7f51ae790-kube-api-access-l548q" (OuterVolumeSpecName: "kube-api-access-l548q") pod "105002ba-65d9-40e5-9c3e-91e7f51ae790" (UID: "105002ba-65d9-40e5-9c3e-91e7f51ae790"). InnerVolumeSpecName "kube-api-access-l548q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:31:03.111198 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.111173    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-token" (OuterVolumeSpecName: "istio-token") pod "105002ba-65d9-40e5-9c3e-91e7f51ae790" (UID: "105002ba-65d9-40e5-9c3e-91e7f51ae790"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:31:03.111645 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.111620    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "105002ba-65d9-40e5-9c3e-91e7f51ae790" (UID: "105002ba-65d9-40e5-9c3e-91e7f51ae790"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:31:03.111645 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.111625    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "105002ba-65d9-40e5-9c3e-91e7f51ae790" (UID: "105002ba-65d9-40e5-9c3e-91e7f51ae790"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:31:03.210663 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.210625    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:31:03.210663 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.210655    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:31:03.210663 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.210664    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:31:03.210663 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.210671    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l548q\" (UniqueName: \"kubernetes.io/projected/105002ba-65d9-40e5-9c3e-91e7f51ae790-kube-api-access-l548q\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:31:03.210935 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.210681    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105002ba-65d9-40e5-9c3e-91e7f51ae790-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:31:03.210935 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.210690    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105002ba-65d9-40e5-9c3e-91e7f51ae790-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:31:03.210935 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.210698    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105002ba-65d9-40e5-9c3e-91e7f51ae790-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:31:03.318517 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.318478    2584 generic.go:356] "Generic (PLEG): container finished" podID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerID="bd90e72e71c18e728200a70a14d4c699368a15575f8091b3959eae52b669e511" exitCode=0
Jul 29 21:31:03.318689 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.318532    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"
Jul 29 21:31:03.318689 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.318537    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6" event={"ID":"105002ba-65d9-40e5-9c3e-91e7f51ae790","Type":"ContainerDied","Data":"bd90e72e71c18e728200a70a14d4c699368a15575f8091b3959eae52b669e511"}
Jul 29 21:31:03.318689 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.318589    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6" event={"ID":"105002ba-65d9-40e5-9c3e-91e7f51ae790","Type":"ContainerDied","Data":"85e748a64a9ab262f3dea7822365f217ad309c992d9e22f89476193aeb570485"}
Jul 29 21:31:03.318689 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.318621    2584 scope.go:117] "RemoveContainer" containerID="bd90e72e71c18e728200a70a14d4c699368a15575f8091b3959eae52b669e511"
Jul 29 21:31:03.327507 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.327487    2584 scope.go:117] "RemoveContainer" containerID="bfe6df1a8f6e585be6d4b753c091a721e0c67484c8b2f0f23c1378937fde68ed"
Jul 29 21:31:03.335033 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.335014    2584 scope.go:117] "RemoveContainer" containerID="bd90e72e71c18e728200a70a14d4c699368a15575f8091b3959eae52b669e511"
Jul 29 21:31:03.335306 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:31:03.335286    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd90e72e71c18e728200a70a14d4c699368a15575f8091b3959eae52b669e511\": container with ID starting with bd90e72e71c18e728200a70a14d4c699368a15575f8091b3959eae52b669e511 not found: ID does not exist" containerID="bd90e72e71c18e728200a70a14d4c699368a15575f8091b3959eae52b669e511"
Jul 29 21:31:03.335351 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.335315    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd90e72e71c18e728200a70a14d4c699368a15575f8091b3959eae52b669e511"} err="failed to get container status \"bd90e72e71c18e728200a70a14d4c699368a15575f8091b3959eae52b669e511\": rpc error: code = NotFound desc = could not find container \"bd90e72e71c18e728200a70a14d4c699368a15575f8091b3959eae52b669e511\": container with ID starting with bd90e72e71c18e728200a70a14d4c699368a15575f8091b3959eae52b669e511 not found: ID does not exist"
Jul 29 21:31:03.335351 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.335335    2584 scope.go:117] "RemoveContainer" containerID="bfe6df1a8f6e585be6d4b753c091a721e0c67484c8b2f0f23c1378937fde68ed"
Jul 29 21:31:03.335582 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:31:03.335564    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe6df1a8f6e585be6d4b753c091a721e0c67484c8b2f0f23c1378937fde68ed\": container with ID starting with bfe6df1a8f6e585be6d4b753c091a721e0c67484c8b2f0f23c1378937fde68ed not found: ID does not exist" containerID="bfe6df1a8f6e585be6d4b753c091a721e0c67484c8b2f0f23c1378937fde68ed"
Jul 29 21:31:03.335641 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.335591    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe6df1a8f6e585be6d4b753c091a721e0c67484c8b2f0f23c1378937fde68ed"} err="failed to get container status \"bfe6df1a8f6e585be6d4b753c091a721e0c67484c8b2f0f23c1378937fde68ed\": rpc error: code = NotFound desc = could not find container \"bfe6df1a8f6e585be6d4b753c091a721e0c67484c8b2f0f23c1378937fde68ed\": container with ID starting with bfe6df1a8f6e585be6d4b753c091a721e0c67484c8b2f0f23c1378937fde68ed not found: ID does not exist"
Jul 29 21:31:03.339251 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.339230    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"]
Jul 29 21:31:03.344366 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:03.344341    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a9af2a72-openshift-default-5bccb9c9c6-mpfd6"]
Jul 29 21:31:04.870561 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:04.870527    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="105002ba-65d9-40e5-9c3e-91e7f51ae790" path="/var/lib/kubelet/pods/105002ba-65d9-40e5-9c3e-91e7f51ae790/volumes"
Jul 29 21:31:25.082062 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:31:25.081984    2584 scope.go:117] "RemoveContainer" containerID="cfb6f049ec011542ebe24883b44b62af3b270421a9b0371db515bb6dbfefa145"
Jul 29 21:33:13.161937 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.161864    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"]
Jul 29 21:33:13.162425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.162227    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerName="istio-proxy"
Jul 29 21:33:13.162425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.162239    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerName="istio-proxy"
Jul 29 21:33:13.162425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.162249    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a27a492-1f97-4b2c-b76f-8fb9e27227fe" containerName="revoke-keys"
Jul 29 21:33:13.162425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.162256    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a27a492-1f97-4b2c-b76f-8fb9e27227fe" containerName="revoke-keys"
Jul 29 21:33:13.162425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.162265    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerName="istio-proxy"
Jul 29 21:33:13.162425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.162270    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerName="istio-proxy"
Jul 29 21:33:13.162425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.162328    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerName="istio-proxy"
Jul 29 21:33:13.162425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.162336    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerName="istio-proxy"
Jul 29 21:33:13.162425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.162343    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a27a492-1f97-4b2c-b76f-8fb9e27227fe" containerName="revoke-keys"
Jul 29 21:33:13.162425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.162349    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd4f6f7-13cf-42c2-b238-7eed76a26b2f" containerName="istio-proxy"
Jul 29 21:33:13.165131 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.165115    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.167284 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.167262    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fd48e5e0-openshift-default-dockercfg-658pt\""
Jul 29 21:33:13.181572 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.181542    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"]
Jul 29 21:33:13.291730 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.291688    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-envoy\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.291730 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.291735    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-data\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.291948 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.291753    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-token\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.291948 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.291775    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-podinfo\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.291948 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.291799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fl52\" (UniqueName: \"kubernetes.io/projected/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-kube-api-access-6fl52\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.291948 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.291880    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-workload-socket\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.291948 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.291912    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-workload-certs\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.291948 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.291928    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istiod-ca-cert\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.292165 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.292028    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-credential-socket\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.393036 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393003    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-data\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.393036 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393038    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-token\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.393298 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393066    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-podinfo\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.393298 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393083    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fl52\" (UniqueName: \"kubernetes.io/projected/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-kube-api-access-6fl52\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.393298 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393109    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-workload-socket\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.393298 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393137    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-workload-certs\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.393298 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393181    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istiod-ca-cert\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.393298 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393228    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-credential-socket\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.393298 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393252    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-envoy\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.393633 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393569    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-workload-socket\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.393695 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393665    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-data\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.393807 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393786    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-workload-certs\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.393866 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393810    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-credential-socket\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.394011 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.393988    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istiod-ca-cert\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.395500 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.395482    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-envoy\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.395693 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.395677    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-podinfo\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.400477 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.400455    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-token\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.400698 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.400678    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fl52\" (UniqueName: \"kubernetes.io/projected/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-kube-api-access-6fl52\") pod \"e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") " pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.475592 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.475494    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:13.611057 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.611023    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"]
Jul 29 21:33:13.613349 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:33:13.613320    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7690a2_181a_43ca_8ae8_6f8c91c6ebcf.slice/crio-eb6300eb06ee616f628544dfce555005e987753749afdd9648c2eb1bba957600 WatchSource:0}: Error finding container eb6300eb06ee616f628544dfce555005e987753749afdd9648c2eb1bba957600: Status 404 returned error can't find the container with id eb6300eb06ee616f628544dfce555005e987753749afdd9648c2eb1bba957600
Jul 29 21:33:13.615599 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.615563    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:33:13.615687 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.615646    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:33:13.615735 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.615683    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:33:13.884749 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.884710    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4" event={"ID":"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf","Type":"ContainerStarted","Data":"3da3bdab3a2cc74a9edee2e6b5c80cd904c3ec928de036cfb16709ae9fb50d4e"}
Jul 29 21:33:13.884925 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.884767    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4" event={"ID":"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf","Type":"ContainerStarted","Data":"eb6300eb06ee616f628544dfce555005e987753749afdd9648c2eb1bba957600"}
Jul 29 21:33:13.907372 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:13.907317    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4" podStartSLOduration=0.907301457 podStartE2EDuration="907.301457ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:33:13.904686191 +0000 UTC m=+2809.500977875" watchObservedRunningTime="2026-07-29 21:33:13.907301457 +0000 UTC m=+2809.503593141"
Jul 29 21:33:14.476636 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:14.476597    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:14.481653 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:14.481631    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:14.895816 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:14.895777    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:14.896915 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:14.896893    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:34.986610 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:34.986573    2584 generic.go:356] "Generic (PLEG): container finished" podID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" containerID="3da3bdab3a2cc74a9edee2e6b5c80cd904c3ec928de036cfb16709ae9fb50d4e" exitCode=0
Jul 29 21:33:34.986610 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:34.986612    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4" event={"ID":"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf","Type":"ContainerDied","Data":"3da3bdab3a2cc74a9edee2e6b5c80cd904c3ec928de036cfb16709ae9fb50d4e"}
Jul 29 21:33:34.987083 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:34.986964    2584 scope.go:117] "RemoveContainer" containerID="3da3bdab3a2cc74a9edee2e6b5c80cd904c3ec928de036cfb16709ae9fb50d4e"
Jul 29 21:33:34.987908 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:34.987882    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:33:34.988000 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:34.987950    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:33:34.988000 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:34.987982    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:33:35.997109 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:35.997068    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4" event={"ID":"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf","Type":"ContainerStarted","Data":"b276386320f5817ecea3157448842539ebee5bb55d8c38104bf646e083b38c59"}
Jul 29 21:33:36.476285 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:36.476247    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:36.476521 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:36.476498    2584 patch_prober.go:28] interesting pod/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 29 21:33:36.476574 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:36.476545    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4" podUID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 29 21:33:37.476092 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:37.476055    2584 patch_prober.go:28] interesting pod/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 29 21:33:37.476551 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:37.476107    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4" podUID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 29 21:33:38.476101 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:38.476063    2584 patch_prober.go:28] interesting pod/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 29 21:33:38.476506 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:38.476118    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4" podUID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 29 21:33:39.476733 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:39.476691    2584 patch_prober.go:28] interesting pod/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 29 21:33:39.477126 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:39.476751    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4" podUID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 29 21:33:40.479773 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:40.479741    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:40.480192 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:40.480102    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:40.480674 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:40.480655    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:33:44.242903 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.242871    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"]
Jul 29 21:33:44.243313 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.243230    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerName="istio-proxy"
Jul 29 21:33:44.243313 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.243246    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="105002ba-65d9-40e5-9c3e-91e7f51ae790" containerName="istio-proxy"
Jul 29 21:33:44.246300 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.246275    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.249093 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.249069    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fd48e5e0\"/\"openshift-service-ca.crt\""
Jul 29 21:33:44.249215 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.249090    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fd48e5e0\"/\"e2e-lifecycle-model-fd48e5e0-kserve-self-signed-certs\""
Jul 29 21:33:44.249297 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.249231    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fd48e5e0\"/\"kube-root-ca.crt\""
Jul 29 21:33:44.249362 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.249333    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fd48e5e0\"/\"default-dockercfg-zczgm\""
Jul 29 21:33:44.255591 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.255566    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"]
Jul 29 21:33:44.379914 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.379876    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-tmp-dir\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.380095 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.379932    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcnpm\" (UniqueName: \"kubernetes.io/projected/e9f8f349-135d-4c98-8947-f2852ddca3e1-kube-api-access-gcnpm\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.380095 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.379987    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-model-cache\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.380095 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.380008    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-dshm\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.380095 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.380056    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f8f349-135d-4c98-8947-f2852ddca3e1-tls-certs\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.380324 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.380098    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-home\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.481269 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.481221    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-home\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.481269 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.481275    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-tmp-dir\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.481508 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.481314    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnpm\" (UniqueName: \"kubernetes.io/projected/e9f8f349-135d-4c98-8947-f2852ddca3e1-kube-api-access-gcnpm\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.481508 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.481349    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-model-cache\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.481508 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.481372    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-dshm\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.481646 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.481509    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f8f349-135d-4c98-8947-f2852ddca3e1-tls-certs\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.481702 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.481683    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-home\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.481780 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.481757    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-tmp-dir\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.481898 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.481877    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-model-cache\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.483582 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.483552    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-dshm\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.483812 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.483795    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f8f349-135d-4c98-8947-f2852ddca3e1-tls-certs\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.489193 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.489168    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcnpm\" (UniqueName: \"kubernetes.io/projected/e9f8f349-135d-4c98-8947-f2852ddca3e1-kube-api-access-gcnpm\") pod \"e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") " pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.556654 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.556556    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:44.691281 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:44.691255    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"]
Jul 29 21:33:45.042075 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:45.042026    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6" event={"ID":"e9f8f349-135d-4c98-8947-f2852ddca3e1","Type":"ContainerStarted","Data":"cfacd4729eab46ce76a58dca89403f27e1bf7f73a467788d98bcd94a82b9cca1"}
Jul 29 21:33:45.042282 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:45.042085    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6" event={"ID":"e9f8f349-135d-4c98-8947-f2852ddca3e1","Type":"ContainerStarted","Data":"388a8c8cba8511a67ce81718ee37b74ead85dc8bf781f8efafe29de893696220"}
Jul 29 21:33:45.057899 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:45.057852    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6" podStartSLOduration=1.05783718 podStartE2EDuration="1.05783718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:33:45.056456331 +0000 UTC m=+2840.652748006" watchObservedRunningTime="2026-07-29 21:33:45.05783718 +0000 UTC m=+2840.654128869"
Jul 29 21:33:54.557216 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:54.557178    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:54.557620 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:54.557236    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:54.569661 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:54.569633    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:33:55.102836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:33:55.102806    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:34:20.569898 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:20.569860    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k"]
Jul 29 21:34:20.573872 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:20.573850    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k"
Jul 29 21:34:20.575969 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:20.575950    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\""
Jul 29 21:34:20.576041 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:20.575991    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:34:20.582574 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:20.582544    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k"]
Jul 29 21:34:20.591758 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:20.591727    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b2932d-e832-403e-9576-7bb4b703a1c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k\" (UID: \"18b2932d-e832-403e-9576-7bb4b703a1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k"
Jul 29 21:34:20.693006 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:20.692959    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b2932d-e832-403e-9576-7bb4b703a1c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k\" (UID: \"18b2932d-e832-403e-9576-7bb4b703a1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k"
Jul 29 21:34:20.693689 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:20.693664    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b2932d-e832-403e-9576-7bb4b703a1c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k\" (UID: \"18b2932d-e832-403e-9576-7bb4b703a1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k"
Jul 29 21:34:20.886168 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:20.886113    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k"
Jul 29 21:34:21.022387 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:21.022361    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k"]
Jul 29 21:34:21.024806 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:34:21.024782    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b2932d_e832_403e_9576_7bb4b703a1c0.slice/crio-bfd5197cd72b970266e44602a0fb9747d7cdde072bcc9a3282cb2f21ebb78b98 WatchSource:0}: Error finding container bfd5197cd72b970266e44602a0fb9747d7cdde072bcc9a3282cb2f21ebb78b98: Status 404 returned error can't find the container with id bfd5197cd72b970266e44602a0fb9747d7cdde072bcc9a3282cb2f21ebb78b98
Jul 29 21:34:21.211809 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:21.211775    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k" event={"ID":"18b2932d-e832-403e-9576-7bb4b703a1c0","Type":"ContainerStarted","Data":"bfd5197cd72b970266e44602a0fb9747d7cdde072bcc9a3282cb2f21ebb78b98"}
Jul 29 21:34:22.218218 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:22.218184    2584 generic.go:356] "Generic (PLEG): container finished" podID="18b2932d-e832-403e-9576-7bb4b703a1c0" containerID="76e4679203db06bc5cc529d068f9f4443cc154d72e339f8b5e17dc125539a037" exitCode=0
Jul 29 21:34:22.218218 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:22.218220    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k" event={"ID":"18b2932d-e832-403e-9576-7bb4b703a1c0","Type":"ContainerDied","Data":"76e4679203db06bc5cc529d068f9f4443cc154d72e339f8b5e17dc125539a037"}
Jul 29 21:34:23.354306 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:23.354284    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k"
Jul 29 21:34:23.417016 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:23.416984    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b2932d-e832-403e-9576-7bb4b703a1c0-maas-api-service-ca\") pod \"18b2932d-e832-403e-9576-7bb4b703a1c0\" (UID: \"18b2932d-e832-403e-9576-7bb4b703a1c0\") "
Jul 29 21:34:23.417397 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:23.417373    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b2932d-e832-403e-9576-7bb4b703a1c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18b2932d-e832-403e-9576-7bb4b703a1c0" (UID: "18b2932d-e832-403e-9576-7bb4b703a1c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:34:23.518491 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:23.518414    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b2932d-e832-403e-9576-7bb4b703a1c0-maas-api-service-ca\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:34:24.229127 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:24.229101    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k"
Jul 29 21:34:24.229369 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:24.229131    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k" event={"ID":"18b2932d-e832-403e-9576-7bb4b703a1c0","Type":"ContainerDied","Data":"bfd5197cd72b970266e44602a0fb9747d7cdde072bcc9a3282cb2f21ebb78b98"}
Jul 29 21:34:24.229369 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:24.229204    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfd5197cd72b970266e44602a0fb9747d7cdde072bcc9a3282cb2f21ebb78b98"
Jul 29 21:34:36.238321 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:36.238288    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k"]
Jul 29 21:34:36.241041 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:36.241018    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fd48e5e0-2pl9k"]
Jul 29 21:34:36.865251 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:36.865220    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18b2932d-e832-403e-9576-7bb4b703a1c0" path="/var/lib/kubelet/pods/18b2932d-e832-403e-9576-7bb4b703a1c0/volumes"
Jul 29 21:34:44.330197 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:44.330164    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"]
Jul 29 21:34:59.348932 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.348865    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6" podUID="e9f8f349-135d-4c98-8947-f2852ddca3e1" containerName="main" containerID="cri-o://cfacd4729eab46ce76a58dca89403f27e1bf7f73a467788d98bcd94a82b9cca1" gracePeriod=45
Jul 29 21:34:59.596655 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.596634    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:34:59.714001 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.713923    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-home\") pod \"e9f8f349-135d-4c98-8947-f2852ddca3e1\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") "
Jul 29 21:34:59.714001 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.713979    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcnpm\" (UniqueName: \"kubernetes.io/projected/e9f8f349-135d-4c98-8947-f2852ddca3e1-kube-api-access-gcnpm\") pod \"e9f8f349-135d-4c98-8947-f2852ddca3e1\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") "
Jul 29 21:34:59.714001 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.713995    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f8f349-135d-4c98-8947-f2852ddca3e1-tls-certs\") pod \"e9f8f349-135d-4c98-8947-f2852ddca3e1\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") "
Jul 29 21:34:59.714291 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.714029    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-dshm\") pod \"e9f8f349-135d-4c98-8947-f2852ddca3e1\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") "
Jul 29 21:34:59.714291 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.714058    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-tmp-dir\") pod \"e9f8f349-135d-4c98-8947-f2852ddca3e1\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") "
Jul 29 21:34:59.714291 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.714075    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-model-cache\") pod \"e9f8f349-135d-4c98-8947-f2852ddca3e1\" (UID: \"e9f8f349-135d-4c98-8947-f2852ddca3e1\") "
Jul 29 21:34:59.714291 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.714238    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-home" (OuterVolumeSpecName: "home") pod "e9f8f349-135d-4c98-8947-f2852ddca3e1" (UID: "e9f8f349-135d-4c98-8947-f2852ddca3e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:34:59.714500 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.714342    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:34:59.714500 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.714444    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9f8f349-135d-4c98-8947-f2852ddca3e1" (UID: "e9f8f349-135d-4c98-8947-f2852ddca3e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:34:59.714500 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.714477    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9f8f349-135d-4c98-8947-f2852ddca3e1" (UID: "e9f8f349-135d-4c98-8947-f2852ddca3e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:34:59.716189 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.716169    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-dshm" (OuterVolumeSpecName: "dshm") pod "e9f8f349-135d-4c98-8947-f2852ddca3e1" (UID: "e9f8f349-135d-4c98-8947-f2852ddca3e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:34:59.716273 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.716169    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f8f349-135d-4c98-8947-f2852ddca3e1-kube-api-access-gcnpm" (OuterVolumeSpecName: "kube-api-access-gcnpm") pod "e9f8f349-135d-4c98-8947-f2852ddca3e1" (UID: "e9f8f349-135d-4c98-8947-f2852ddca3e1"). InnerVolumeSpecName "kube-api-access-gcnpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:34:59.716314 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.716267    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f8f349-135d-4c98-8947-f2852ddca3e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9f8f349-135d-4c98-8947-f2852ddca3e1" (UID: "e9f8f349-135d-4c98-8947-f2852ddca3e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:34:59.814826 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.814789    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:34:59.814826 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.814821    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:34:59.814826 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.814831    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f8f349-135d-4c98-8947-f2852ddca3e1-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:34:59.815070 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.814839    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcnpm\" (UniqueName: \"kubernetes.io/projected/e9f8f349-135d-4c98-8947-f2852ddca3e1-kube-api-access-gcnpm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:34:59.815070 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:34:59.814849    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f8f349-135d-4c98-8947-f2852ddca3e1-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:35:00.393783 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:00.393739    2584 generic.go:356] "Generic (PLEG): container finished" podID="e9f8f349-135d-4c98-8947-f2852ddca3e1" containerID="cfacd4729eab46ce76a58dca89403f27e1bf7f73a467788d98bcd94a82b9cca1" exitCode=0
Jul 29 21:35:00.393783 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:00.393781    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"
Jul 29 21:35:00.394350 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:00.393823    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6" event={"ID":"e9f8f349-135d-4c98-8947-f2852ddca3e1","Type":"ContainerDied","Data":"cfacd4729eab46ce76a58dca89403f27e1bf7f73a467788d98bcd94a82b9cca1"}
Jul 29 21:35:00.394350 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:00.393872    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6" event={"ID":"e9f8f349-135d-4c98-8947-f2852ddca3e1","Type":"ContainerDied","Data":"388a8c8cba8511a67ce81718ee37b74ead85dc8bf781f8efafe29de893696220"}
Jul 29 21:35:00.394350 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:00.393897    2584 scope.go:117] "RemoveContainer" containerID="cfacd4729eab46ce76a58dca89403f27e1bf7f73a467788d98bcd94a82b9cca1"
Jul 29 21:35:00.402497 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:00.402481    2584 scope.go:117] "RemoveContainer" containerID="cfacd4729eab46ce76a58dca89403f27e1bf7f73a467788d98bcd94a82b9cca1"
Jul 29 21:35:00.402731 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:35:00.402710    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfacd4729eab46ce76a58dca89403f27e1bf7f73a467788d98bcd94a82b9cca1\": container with ID starting with cfacd4729eab46ce76a58dca89403f27e1bf7f73a467788d98bcd94a82b9cca1 not found: ID does not exist" containerID="cfacd4729eab46ce76a58dca89403f27e1bf7f73a467788d98bcd94a82b9cca1"
Jul 29 21:35:00.402777 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:00.402740    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfacd4729eab46ce76a58dca89403f27e1bf7f73a467788d98bcd94a82b9cca1"} err="failed to get container status \"cfacd4729eab46ce76a58dca89403f27e1bf7f73a467788d98bcd94a82b9cca1\": rpc error: code = NotFound desc = could not find container \"cfacd4729eab46ce76a58dca89403f27e1bf7f73a467788d98bcd94a82b9cca1\": container with ID starting with cfacd4729eab46ce76a58dca89403f27e1bf7f73a467788d98bcd94a82b9cca1 not found: ID does not exist"
Jul 29 21:35:00.416672 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:00.413430    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"]
Jul 29 21:35:00.419805 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:00.419774    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-fd48e5e0/e2e-lifecycle-model-fd48e5e0-kserve-577bcd9955-jw8s6"]
Jul 29 21:35:00.865669 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:00.865636    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9f8f349-135d-4c98-8947-f2852ddca3e1" path="/var/lib/kubelet/pods/e9f8f349-135d-4c98-8947-f2852ddca3e1/volumes"
Jul 29 21:35:27.315816 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:27.315783    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"]
Jul 29 21:35:27.316264 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:27.315984    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4" podUID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" containerName="istio-proxy" containerID="cri-o://b276386320f5817ecea3157448842539ebee5bb55d8c38104bf646e083b38c59" gracePeriod=30
Jul 29 21:35:32.539070 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.539039    2584 generic.go:356] "Generic (PLEG): container finished" podID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" containerID="b276386320f5817ecea3157448842539ebee5bb55d8c38104bf646e083b38c59" exitCode=0
Jul 29 21:35:32.539527 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.539108    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4" event={"ID":"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf","Type":"ContainerDied","Data":"b276386320f5817ecea3157448842539ebee5bb55d8c38104bf646e083b38c59"}
Jul 29 21:35:32.539527 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.539186    2584 scope.go:117] "RemoveContainer" containerID="3da3bdab3a2cc74a9edee2e6b5c80cd904c3ec928de036cfb16709ae9fb50d4e"
Jul 29 21:35:32.569316 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.569293    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:35:32.681230 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681202    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-data\") pod \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") "
Jul 29 21:35:32.681378 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681237    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-workload-certs\") pod \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") "
Jul 29 21:35:32.681378 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681295    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-envoy\") pod \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") "
Jul 29 21:35:32.681378 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681345    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istiod-ca-cert\") pod \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") "
Jul 29 21:35:32.681378 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681371    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-credential-socket\") pod \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") "
Jul 29 21:35:32.681561 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681398    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-podinfo\") pod \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") "
Jul 29 21:35:32.681561 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681422    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-token\") pod \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") "
Jul 29 21:35:32.681561 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681445    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fl52\" (UniqueName: \"kubernetes.io/projected/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-kube-api-access-6fl52\") pod \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") "
Jul 29 21:35:32.681561 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681469    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-workload-socket\") pod \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\" (UID: \"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf\") "
Jul 29 21:35:32.681561 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681551    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" (UID: "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:35:32.681806 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681734    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:35:32.681806 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681759    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" (UID: "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:35:32.681896 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681795    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" (UID: "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:35:32.681896 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.681846    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" (UID: "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:35:32.682181 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.682138    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-data" (OuterVolumeSpecName: "istio-data") pod "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" (UID: "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:35:32.683491 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.683466    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" (UID: "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:35:32.683729 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.683707    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-kube-api-access-6fl52" (OuterVolumeSpecName: "kube-api-access-6fl52") pod "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" (UID: "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf"). InnerVolumeSpecName "kube-api-access-6fl52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:35:32.683998 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.683971    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" (UID: "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:35:32.683998 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.683984    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-token" (OuterVolumeSpecName: "istio-token") pod "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" (UID: "2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:35:32.782647 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.782612    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:35:32.782647 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.782644    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:35:32.782797 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.782661    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:35:32.782797 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.782674    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:35:32.782797 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.782685    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fl52\" (UniqueName: \"kubernetes.io/projected/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-kube-api-access-6fl52\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:35:32.782797 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.782699    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:35:32.782797 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.782711    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:35:32.782797 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:32.782723    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:35:33.544636 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:33.544593    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4" event={"ID":"2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf","Type":"ContainerDied","Data":"eb6300eb06ee616f628544dfce555005e987753749afdd9648c2eb1bba957600"}
Jul 29 21:35:33.545045 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:33.544647    2584 scope.go:117] "RemoveContainer" containerID="b276386320f5817ecea3157448842539ebee5bb55d8c38104bf646e083b38c59"
Jul 29 21:35:33.545045 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:33.544607    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"
Jul 29 21:35:33.562966 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:33.562944    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"]
Jul 29 21:35:33.566792 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:33.566770    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fd48e5e0-openshift-default-856f49bbbd-c9ch4"]
Jul 29 21:35:34.865098 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:34.865063    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" path="/var/lib/kubelet/pods/2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf/volumes"
Jul 29 21:35:39.466796 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.466762    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"]
Jul 29 21:35:39.467176 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.467054    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f8f349-135d-4c98-8947-f2852ddca3e1" containerName="main"
Jul 29 21:35:39.467176 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.467064    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f8f349-135d-4c98-8947-f2852ddca3e1" containerName="main"
Jul 29 21:35:39.467176 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.467075    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" containerName="istio-proxy"
Jul 29 21:35:39.467176 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.467081    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" containerName="istio-proxy"
Jul 29 21:35:39.467176 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.467089    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" containerName="istio-proxy"
Jul 29 21:35:39.467176 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.467094    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" containerName="istio-proxy"
Jul 29 21:35:39.467176 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.467109    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b2932d-e832-403e-9576-7bb4b703a1c0" containerName="revoke-keys"
Jul 29 21:35:39.467176 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.467114    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b2932d-e832-403e-9576-7bb4b703a1c0" containerName="revoke-keys"
Jul 29 21:35:39.467462 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.467204    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b2932d-e832-403e-9576-7bb4b703a1c0" containerName="revoke-keys"
Jul 29 21:35:39.467462 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.467214    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9f8f349-135d-4c98-8947-f2852ddca3e1" containerName="main"
Jul 29 21:35:39.467462 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.467221    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" containerName="istio-proxy"
Jul 29 21:35:39.467462 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.467226    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7690a2-181a-43ca-8ae8-6f8c91c6ebcf" containerName="istio-proxy"
Jul 29 21:35:39.473814 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.473783    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.478573 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.478554    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bdcc5a3e-openshift-default-dockercfg-tmkqc\""
Jul 29 21:35:39.484620 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.484588    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"]
Jul 29 21:35:39.535207 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.535173    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-workload-certs\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.535351 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.535225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-workload-socket\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.535351 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.535284    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd46a33-e08e-4cb9-9227-74911a409720-istiod-ca-cert\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.535351 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.535307    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd46a33-e08e-4cb9-9227-74911a409720-istio-podinfo\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.535517 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.535361    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-istio-envoy\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.535517 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.535396    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-credential-socket\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.535517 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.535424    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd46a33-e08e-4cb9-9227-74911a409720-istio-token\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.535517 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.535465    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjkq4\" (UniqueName: \"kubernetes.io/projected/cfd46a33-e08e-4cb9-9227-74911a409720-kube-api-access-tjkq4\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.535517 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.535504    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-istio-data\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.636475 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.636434    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-istio-data\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.636668 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.636490    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-workload-certs\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.636668 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.636534    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-workload-socket\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.636668 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.636574    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd46a33-e08e-4cb9-9227-74911a409720-istiod-ca-cert\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.636668 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.636597    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd46a33-e08e-4cb9-9227-74911a409720-istio-podinfo\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.636668 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.636658    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-istio-envoy\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.636928 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.636685    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-credential-socket\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.636928 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.636716    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd46a33-e08e-4cb9-9227-74911a409720-istio-token\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.636928 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.636782    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjkq4\" (UniqueName: \"kubernetes.io/projected/cfd46a33-e08e-4cb9-9227-74911a409720-kube-api-access-tjkq4\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.637087 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.636928    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-workload-certs\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.637087 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.636965    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-workload-socket\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.637087 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.637024    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-istio-data\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.637285 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.637261    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-credential-socket\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.637723 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.637697    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd46a33-e08e-4cb9-9227-74911a409720-istiod-ca-cert\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.639228 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.639208    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-istio-envoy\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.639373 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.639356    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd46a33-e08e-4cb9-9227-74911a409720-istio-podinfo\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.644740 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.644718    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjkq4\" (UniqueName: \"kubernetes.io/projected/cfd46a33-e08e-4cb9-9227-74911a409720-kube-api-access-tjkq4\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.645599 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.645569    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd46a33-e08e-4cb9-9227-74911a409720-istio-token\") pod \"e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") " pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.784611 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.784530    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:39.914946 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.914921    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"]
Jul 29 21:35:39.917217 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:35:39.917194    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd46a33_e08e_4cb9_9227_74911a409720.slice/crio-dae8cbb92cb2b75082c8b39428f99959f4fdb8285f2457e92d9b0836a19f3b17 WatchSource:0}: Error finding container dae8cbb92cb2b75082c8b39428f99959f4fdb8285f2457e92d9b0836a19f3b17: Status 404 returned error can't find the container with id dae8cbb92cb2b75082c8b39428f99959f4fdb8285f2457e92d9b0836a19f3b17
Jul 29 21:35:39.919415 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.919378    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:35:39.919490 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.919462    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:35:39.919529 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:39.919503    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:35:40.577049 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:40.577012    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" event={"ID":"cfd46a33-e08e-4cb9-9227-74911a409720","Type":"ContainerStarted","Data":"7272c0ab903b037fbbf4372526c52b110164b0eeead11498a785c3e3bc813228"}
Jul 29 21:35:40.577443 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:40.577063    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" event={"ID":"cfd46a33-e08e-4cb9-9227-74911a409720","Type":"ContainerStarted","Data":"dae8cbb92cb2b75082c8b39428f99959f4fdb8285f2457e92d9b0836a19f3b17"}
Jul 29 21:35:40.597199 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:40.597103    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" podStartSLOduration=1.597086596 podStartE2EDuration="1.597086596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:35:40.59574802 +0000 UTC m=+2956.192039704" watchObservedRunningTime="2026-07-29 21:35:40.597086596 +0000 UTC m=+2956.193378279"
Jul 29 21:35:40.784992 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:40.784961    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:40.789591 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:40.789566    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:41.581783 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:41.581752    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:41.582708 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:41.582690    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:56.583469 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:56.583436    2584 patch_prober.go:28] interesting pod/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 29 21:35:56.583866 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:56.583491    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 29 21:35:56.654669 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:56.654638    2584 generic.go:356] "Generic (PLEG): container finished" podID="cfd46a33-e08e-4cb9-9227-74911a409720" containerID="7272c0ab903b037fbbf4372526c52b110164b0eeead11498a785c3e3bc813228" exitCode=0
Jul 29 21:35:56.654821 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:56.654715    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" event={"ID":"cfd46a33-e08e-4cb9-9227-74911a409720","Type":"ContainerDied","Data":"7272c0ab903b037fbbf4372526c52b110164b0eeead11498a785c3e3bc813228"}
Jul 29 21:35:56.655127 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:56.655112    2584 scope.go:117] "RemoveContainer" containerID="7272c0ab903b037fbbf4372526c52b110164b0eeead11498a785c3e3bc813228"
Jul 29 21:35:56.656127 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:56.656101    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:35:56.656239 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:56.656221    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:35:56.656283 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:56.656270    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:35:57.660602 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:57.660567    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" event={"ID":"cfd46a33-e08e-4cb9-9227-74911a409720","Type":"ContainerStarted","Data":"897cc53f68a9e0d2462ac2ee86cc3c331eea63ee7c973fd8c539becb74ef1a92"}
Jul 29 21:35:57.784957 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:57.784924    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:35:57.785232 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:57.785208    2584 patch_prober.go:28] interesting pod/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 29 21:35:57.785292 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:57.785256    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 29 21:35:58.784811 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:58.784779    2584 patch_prober.go:28] interesting pod/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 29 21:35:58.785208 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:58.784825    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 29 21:35:59.785190 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:59.785127    2584 patch_prober.go:28] interesting pod/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 29 21:35:59.785586 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:35:59.785214    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 29 21:36:00.785189 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:00.785125    2584 patch_prober.go:28] interesting pod/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 29 21:36:00.785409 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:00.785223    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 29 21:36:01.785418 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:01.785385    2584 patch_prober.go:28] interesting pod/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 29 21:36:01.785797 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:01.785452    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 29 21:36:02.788772 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:02.788744    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:36:02.789186 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:02.789166    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:36:02.789907 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:02.789888    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:36:18.928478 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:18.928442    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"]
Jul 29 21:36:18.931942 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:18.931921    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:18.934130 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:18.934104    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-641c4002-openshift-default-dockercfg-xbz2s\""
Jul 29 21:36:18.943573 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:18.943552    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"]
Jul 29 21:36:19.070584 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.070551    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-data\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.070584 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.070586    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-token\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.070798 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.070608    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istiod-ca-cert\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.070798 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.070698    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-workload-certs\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.070798 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.070753    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-credential-socket\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.070903 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.070792    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-envoy\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.070903 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.070831    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-workload-socket\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.070903 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.070850    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-podinfo\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.070903 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.070872    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzvm\" (UniqueName: \"kubernetes.io/projected/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-kube-api-access-bkzvm\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.171967 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.171938    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istiod-ca-cert\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.172131 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.171974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-workload-certs\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.172131 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.172007    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-credential-socket\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.172131 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.172036    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-envoy\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.172131 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.172071    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-workload-socket\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.172131 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.172098    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-podinfo\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.172417 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.172241    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzvm\" (UniqueName: \"kubernetes.io/projected/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-kube-api-access-bkzvm\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.172417 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.172295    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-data\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.172417 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.172336    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-token\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.172571 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.172455    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-workload-socket\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.172571 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.172492    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-workload-certs\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.172742 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.172715    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istiod-ca-cert\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.172851 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.172715    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-data\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.172851 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.172779    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-credential-socket\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.174402 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.174376    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-envoy\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.174601 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.174584    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-podinfo\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.180011 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.179962    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-token\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.180483 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.180465    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzvm\" (UniqueName: \"kubernetes.io/projected/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-kube-api-access-bkzvm\") pod \"e2e-mt-641c4002-openshift-default-76f64b9856-b548m\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") " pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.241580 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.241556    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:19.371063 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.371037    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"]
Jul 29 21:36:19.372769 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:36:19.372741    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abbfd11_fd4a_422c_9889_2ea42d4fb06e.slice/crio-0e89aafed738fcd241174efeed5e63769404e7321ff1c4d8af5cca2d79d2665f WatchSource:0}: Error finding container 0e89aafed738fcd241174efeed5e63769404e7321ff1c4d8af5cca2d79d2665f: Status 404 returned error can't find the container with id 0e89aafed738fcd241174efeed5e63769404e7321ff1c4d8af5cca2d79d2665f
Jul 29 21:36:19.374721 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.374689    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:36:19.374788 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.374752    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:36:19.374788 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.374780    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:36:19.764193 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.764131    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m" event={"ID":"2abbfd11-fd4a-422c-9889-2ea42d4fb06e","Type":"ContainerStarted","Data":"f239fcd181e92b870454e95e96dfd85eb6fe603465bb34309ebea70893e37c5b"}
Jul 29 21:36:19.764370 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.764257    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m" event={"ID":"2abbfd11-fd4a-422c-9889-2ea42d4fb06e","Type":"ContainerStarted","Data":"0e89aafed738fcd241174efeed5e63769404e7321ff1c4d8af5cca2d79d2665f"}
Jul 29 21:36:19.782408 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:19.782359    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m" podStartSLOduration=1.782344142 podStartE2EDuration="1.782344142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:36:19.781283294 +0000 UTC m=+2995.377575000" watchObservedRunningTime="2026-07-29 21:36:19.782344142 +0000 UTC m=+2995.378635903"
Jul 29 21:36:20.242080 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:20.242048    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:20.246721 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:20.246699    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:20.768756 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:20.768724    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:20.769548 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:20.769528    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:25.137037 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:25.137003    2584 scope.go:117] "RemoveContainer" containerID="606b30cf4a3ab0c453856f639fe2bd70432ab452b4a677a3f33185b7fa94d0fa"
Jul 29 21:36:25.144675 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:25.144657    2584 scope.go:117] "RemoveContainer" containerID="62e6da4790245534ee1cc41bef80341c9c43ac5bf77e8f4db37f081529f9ae88"
Jul 29 21:36:39.855014 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:39.854981    2584 generic.go:356] "Generic (PLEG): container finished" podID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerID="f239fcd181e92b870454e95e96dfd85eb6fe603465bb34309ebea70893e37c5b" exitCode=0
Jul 29 21:36:39.855453 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:39.855053    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m" event={"ID":"2abbfd11-fd4a-422c-9889-2ea42d4fb06e","Type":"ContainerDied","Data":"f239fcd181e92b870454e95e96dfd85eb6fe603465bb34309ebea70893e37c5b"}
Jul 29 21:36:39.855496 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:39.855478    2584 scope.go:117] "RemoveContainer" containerID="f239fcd181e92b870454e95e96dfd85eb6fe603465bb34309ebea70893e37c5b"
Jul 29 21:36:39.856427 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:39.856403    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:36:39.856495 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:39.856477    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:36:39.856536 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:39.856525    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:36:40.867606 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:40.867570    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m" event={"ID":"2abbfd11-fd4a-422c-9889-2ea42d4fb06e","Type":"ContainerStarted","Data":"deabcdab819f193ca0d35d272786ef1fc731f81934632339225cd1d0c8408118"}
Jul 29 21:36:41.242509 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:41.242411    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:41.242679 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:41.242656    2584 patch_prober.go:28] interesting pod/e2e-mt-641c4002-openshift-default-76f64b9856-b548m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 29 21:36:41.242757 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:41.242725    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m" podUID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 29 21:36:42.242254 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:42.242217    2584 patch_prober.go:28] interesting pod/e2e-mt-641c4002-openshift-default-76f64b9856-b548m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 29 21:36:42.242668 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:42.242277    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m" podUID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 29 21:36:43.242154 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:43.242112    2584 patch_prober.go:28] interesting pod/e2e-mt-641c4002-openshift-default-76f64b9856-b548m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 29 21:36:43.242336 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:43.242198    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m" podUID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 29 21:36:44.242865 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:44.242827    2584 patch_prober.go:28] interesting pod/e2e-mt-641c4002-openshift-default-76f64b9856-b548m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 29 21:36:44.243298 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:44.242884    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m" podUID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 29 21:36:45.246158 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:45.246105    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:45.246649 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:45.246524    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:36:45.247076 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:36:45.247057    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:37:25.168306 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:37:25.168279    2584 scope.go:117] "RemoveContainer" containerID="a2597ba131880aecb80ff494972f63dc5e0fae806491ddebd610bc0190c0e98d"
Jul 29 21:40:02.022670 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:02.022637    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp"]
Jul 29 21:40:02.025916 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:02.025898    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp"
Jul 29 21:40:02.027882 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:02.027860    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:40:02.027993 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:02.027865    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\""
Jul 29 21:40:02.039268 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:02.039239    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp"]
Jul 29 21:40:02.126073 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:02.126031    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610f331e-7595-42dc-831e-4467ca052a9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp\" (UID: \"610f331e-7595-42dc-831e-4467ca052a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp"
Jul 29 21:40:02.226744 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:02.226711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610f331e-7595-42dc-831e-4467ca052a9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp\" (UID: \"610f331e-7595-42dc-831e-4467ca052a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp"
Jul 29 21:40:02.227362 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:02.227343    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610f331e-7595-42dc-831e-4467ca052a9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp\" (UID: \"610f331e-7595-42dc-831e-4467ca052a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp"
Jul 29 21:40:02.335749 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:02.335653    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp"
Jul 29 21:40:02.466809 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:02.466776    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp"]
Jul 29 21:40:02.469326 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:40:02.469298    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610f331e_7595_42dc_831e_4467ca052a9f.slice/crio-bbe8b8bdf5711ddb89ea0e5e23565e5bc2013a1f67680a12325efb56316b9a3d WatchSource:0}: Error finding container bbe8b8bdf5711ddb89ea0e5e23565e5bc2013a1f67680a12325efb56316b9a3d: Status 404 returned error can't find the container with id bbe8b8bdf5711ddb89ea0e5e23565e5bc2013a1f67680a12325efb56316b9a3d
Jul 29 21:40:02.774562 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:02.774529    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp" event={"ID":"610f331e-7595-42dc-831e-4467ca052a9f","Type":"ContainerStarted","Data":"8df35cdb16e305a3031d2363912714c5e56e2c067def3afe34cfea3b15e2c83d"}
Jul 29 21:40:02.774742 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:02.774581    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp" event={"ID":"610f331e-7595-42dc-831e-4467ca052a9f","Type":"ContainerStarted","Data":"bbe8b8bdf5711ddb89ea0e5e23565e5bc2013a1f67680a12325efb56316b9a3d"}
Jul 29 21:40:02.788825 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:02.788780    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp" podStartSLOduration=0.788763587 podStartE2EDuration="788.763587ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:40:02.788012341 +0000 UTC m=+3218.384304025" watchObservedRunningTime="2026-07-29 21:40:02.788763587 +0000 UTC m=+3218.385055272"
Jul 29 21:40:03.782439 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:03.782388    2584 generic.go:356] "Generic (PLEG): container finished" podID="610f331e-7595-42dc-831e-4467ca052a9f" containerID="8df35cdb16e305a3031d2363912714c5e56e2c067def3afe34cfea3b15e2c83d" exitCode=0
Jul 29 21:40:03.782837 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:03.782467    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp" event={"ID":"610f331e-7595-42dc-831e-4467ca052a9f","Type":"ContainerDied","Data":"8df35cdb16e305a3031d2363912714c5e56e2c067def3afe34cfea3b15e2c83d"}
Jul 29 21:40:04.915860 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:04.915837    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp"
Jul 29 21:40:05.050292 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:05.050198    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610f331e-7595-42dc-831e-4467ca052a9f-maas-api-service-ca\") pod \"610f331e-7595-42dc-831e-4467ca052a9f\" (UID: \"610f331e-7595-42dc-831e-4467ca052a9f\") "
Jul 29 21:40:05.050619 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:05.050597    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610f331e-7595-42dc-831e-4467ca052a9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "610f331e-7595-42dc-831e-4467ca052a9f" (UID: "610f331e-7595-42dc-831e-4467ca052a9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:40:05.151381 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:05.151343    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610f331e-7595-42dc-831e-4467ca052a9f-maas-api-service-ca\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:40:05.793880 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:05.793853    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp"
Jul 29 21:40:05.794054 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:05.793891    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp" event={"ID":"610f331e-7595-42dc-831e-4467ca052a9f","Type":"ContainerDied","Data":"bbe8b8bdf5711ddb89ea0e5e23565e5bc2013a1f67680a12325efb56316b9a3d"}
Jul 29 21:40:05.794054 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:05.793941    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbe8b8bdf5711ddb89ea0e5e23565e5bc2013a1f67680a12325efb56316b9a3d"
Jul 29 21:40:25.199534 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:40:25.199458    2584 scope.go:117] "RemoveContainer" containerID="76e4679203db06bc5cc529d068f9f4443cc154d72e339f8b5e17dc125539a037"
Jul 29 21:45:00.140227 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.138085    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn"]
Jul 29 21:45:00.140227 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.138912    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610f331e-7595-42dc-831e-4467ca052a9f" containerName="revoke-keys"
Jul 29 21:45:00.140227 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.138934    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="610f331e-7595-42dc-831e-4467ca052a9f" containerName="revoke-keys"
Jul 29 21:45:00.140227 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.139104    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="610f331e-7595-42dc-831e-4467ca052a9f" containerName="revoke-keys"
Jul 29 21:45:00.142911 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.142888    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn"
Jul 29 21:45:00.144787 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.144762    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756025-9rr4b"]
Jul 29 21:45:00.145188 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.145166    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\""
Jul 29 21:45:00.148239 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.148220    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-9rr4b"
Jul 29 21:45:00.154205 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.154034    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn"
Jul 29 21:45:00.155943 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.155916    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn"]
Jul 29 21:45:00.159575 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.159419    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756025-9rr4b"]
Jul 29 21:45:00.163998 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.163981    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-9rr4b"
Jul 29 21:45:00.300347 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.300314    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn"]
Jul 29 21:45:00.303382 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:45:00.303355    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fb8976_8f8e_48ef_b682_d41f621a29af.slice/crio-1a0cdb84146146ad80044ac1a28c01a354eb7f701aec976ff6a148b52534678d WatchSource:0}: Error finding container 1a0cdb84146146ad80044ac1a28c01a354eb7f701aec976ff6a148b52534678d: Status 404 returned error can't find the container with id 1a0cdb84146146ad80044ac1a28c01a354eb7f701aec976ff6a148b52534678d
Jul 29 21:45:00.326556 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:00.326534    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756025-9rr4b"]
Jul 29 21:45:00.328814 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:45:00.328792    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f453691_7fea_4373_86b0_316dd0c09139.slice/crio-e7dd51e8d3662e4388cdb07342a3aa5871e0337efdc40bdc7393d7eb0b8281e0 WatchSource:0}: Error finding container e7dd51e8d3662e4388cdb07342a3aa5871e0337efdc40bdc7393d7eb0b8281e0: Status 404 returned error can't find the container with id e7dd51e8d3662e4388cdb07342a3aa5871e0337efdc40bdc7393d7eb0b8281e0
Jul 29 21:45:01.108033 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:01.108002    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-9rr4b" event={"ID":"7f453691-7fea-4373-86b0-316dd0c09139","Type":"ContainerStarted","Data":"e6643eadcf264bab2e5e4a0ea5dcf4708c9eca2d2adbaf44c636448d2410c2dd"}
Jul 29 21:45:01.108204 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:01.108067    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-9rr4b" event={"ID":"7f453691-7fea-4373-86b0-316dd0c09139","Type":"ContainerStarted","Data":"e7dd51e8d3662e4388cdb07342a3aa5871e0337efdc40bdc7393d7eb0b8281e0"}
Jul 29 21:45:01.109572 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:01.109549    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn" event={"ID":"98fb8976-8f8e-48ef-b682-d41f621a29af","Type":"ContainerStarted","Data":"a389dc4415922edfd13730c805a004dfed3edb48b95aef7decf93f45aa321822"}
Jul 29 21:45:01.109663 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:01.109592    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn" event={"ID":"98fb8976-8f8e-48ef-b682-d41f621a29af","Type":"ContainerStarted","Data":"1a0cdb84146146ad80044ac1a28c01a354eb7f701aec976ff6a148b52534678d"}
Jul 29 21:45:01.122646 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:01.122606    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-9rr4b" podStartSLOduration=1.122592661 podStartE2EDuration="1.122592661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:45:01.120360046 +0000 UTC m=+3516.716651729" watchObservedRunningTime="2026-07-29 21:45:01.122592661 +0000 UTC m=+3516.718884344"
Jul 29 21:45:01.134552 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:01.134512    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn" podStartSLOduration=1.134497568 podStartE2EDuration="1.134497568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:45:01.133626723 +0000 UTC m=+3516.729918430" watchObservedRunningTime="2026-07-29 21:45:01.134497568 +0000 UTC m=+3516.730789252"
Jul 29 21:45:02.116218 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:02.116186    2584 generic.go:356] "Generic (PLEG): container finished" podID="7f453691-7fea-4373-86b0-316dd0c09139" containerID="e6643eadcf264bab2e5e4a0ea5dcf4708c9eca2d2adbaf44c636448d2410c2dd" exitCode=0
Jul 29 21:45:02.116629 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:02.116281    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-9rr4b" event={"ID":"7f453691-7fea-4373-86b0-316dd0c09139","Type":"ContainerDied","Data":"e6643eadcf264bab2e5e4a0ea5dcf4708c9eca2d2adbaf44c636448d2410c2dd"}
Jul 29 21:45:02.117719 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:02.117695    2584 generic.go:356] "Generic (PLEG): container finished" podID="98fb8976-8f8e-48ef-b682-d41f621a29af" containerID="a389dc4415922edfd13730c805a004dfed3edb48b95aef7decf93f45aa321822" exitCode=0
Jul 29 21:45:02.117838 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:02.117721    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn" event={"ID":"98fb8976-8f8e-48ef-b682-d41f621a29af","Type":"ContainerDied","Data":"a389dc4415922edfd13730c805a004dfed3edb48b95aef7decf93f45aa321822"}
Jul 29 21:45:02.621139 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:02.621106    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"]
Jul 29 21:45:02.621354 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:02.621330    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy" containerID="cri-o://897cc53f68a9e0d2462ac2ee86cc3c331eea63ee7c973fd8c539becb74ef1a92" gracePeriod=30
Jul 29 21:45:02.791560 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:02.791518    2584 patch_prober.go:28] interesting pod/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 21:45:02.791718 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:02.791579    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:45:03.257763 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:03.257735    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn"
Jul 29 21:45:03.278031 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:03.278008    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-9rr4b"
Jul 29 21:45:04.128459 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:04.128434    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn"
Jul 29 21:45:04.128459 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:04.128445    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn" event={"ID":"98fb8976-8f8e-48ef-b682-d41f621a29af","Type":"ContainerDied","Data":"1a0cdb84146146ad80044ac1a28c01a354eb7f701aec976ff6a148b52534678d"}
Jul 29 21:45:04.128688 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:04.128499    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a0cdb84146146ad80044ac1a28c01a354eb7f701aec976ff6a148b52534678d"
Jul 29 21:45:04.129777 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:04.129755    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-9rr4b" event={"ID":"7f453691-7fea-4373-86b0-316dd0c09139","Type":"ContainerDied","Data":"e7dd51e8d3662e4388cdb07342a3aa5871e0337efdc40bdc7393d7eb0b8281e0"}
Jul 29 21:45:04.129777 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:04.129779    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-9rr4b"
Jul 29 21:45:04.129975 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:04.129787    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7dd51e8d3662e4388cdb07342a3aa5871e0337efdc40bdc7393d7eb0b8281e0"
Jul 29 21:45:04.337731 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:04.337692    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z"]
Jul 29 21:45:04.340293 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:04.340268    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755980-hhp8z"]
Jul 29 21:45:04.864993 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:04.864958    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0444bdab-349d-4148-b08c-f588110c750d" path="/var/lib/kubelet/pods/0444bdab-349d-4148-b08c-f588110c750d/volumes"
Jul 29 21:45:05.050443 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:05.050409    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp"]
Jul 29 21:45:05.053452 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:05.053426    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdcc5a3e-9knxp"]
Jul 29 21:45:06.865336 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:06.865304    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="610f331e-7595-42dc-831e-4467ca052a9f" path="/var/lib/kubelet/pods/610f331e-7595-42dc-831e-4467ca052a9f/volumes"
Jul 29 21:45:07.873013 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:07.872992    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:45:08.012969 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.012873    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-workload-certs\") pod \"cfd46a33-e08e-4cb9-9227-74911a409720\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") "
Jul 29 21:45:08.012969 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.012911    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd46a33-e08e-4cb9-9227-74911a409720-istiod-ca-cert\") pod \"cfd46a33-e08e-4cb9-9227-74911a409720\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") "
Jul 29 21:45:08.012969 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.012961    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-workload-socket\") pod \"cfd46a33-e08e-4cb9-9227-74911a409720\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") "
Jul 29 21:45:08.013422 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.012976    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-istio-data\") pod \"cfd46a33-e08e-4cb9-9227-74911a409720\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") "
Jul 29 21:45:08.013422 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.012996    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-credential-socket\") pod \"cfd46a33-e08e-4cb9-9227-74911a409720\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") "
Jul 29 21:45:08.013422 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.013023    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd46a33-e08e-4cb9-9227-74911a409720-istio-podinfo\") pod \"cfd46a33-e08e-4cb9-9227-74911a409720\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") "
Jul 29 21:45:08.013422 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.013073    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd46a33-e08e-4cb9-9227-74911a409720-istio-token\") pod \"cfd46a33-e08e-4cb9-9227-74911a409720\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") "
Jul 29 21:45:08.013422 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.013099    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjkq4\" (UniqueName: \"kubernetes.io/projected/cfd46a33-e08e-4cb9-9227-74911a409720-kube-api-access-tjkq4\") pod \"cfd46a33-e08e-4cb9-9227-74911a409720\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") "
Jul 29 21:45:08.013422 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.013139    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-istio-envoy\") pod \"cfd46a33-e08e-4cb9-9227-74911a409720\" (UID: \"cfd46a33-e08e-4cb9-9227-74911a409720\") "
Jul 29 21:45:08.013422 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.013192    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cfd46a33-e08e-4cb9-9227-74911a409720" (UID: "cfd46a33-e08e-4cb9-9227-74911a409720"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:08.013422 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.013322    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cfd46a33-e08e-4cb9-9227-74911a409720" (UID: "cfd46a33-e08e-4cb9-9227-74911a409720"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:08.013836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.013420    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd46a33-e08e-4cb9-9227-74911a409720-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cfd46a33-e08e-4cb9-9227-74911a409720" (UID: "cfd46a33-e08e-4cb9-9227-74911a409720"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:45:08.013836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.013508    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:08.013836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.013528    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd46a33-e08e-4cb9-9227-74911a409720-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:08.013836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.013541    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:08.013836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.013638    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cfd46a33-e08e-4cb9-9227-74911a409720" (UID: "cfd46a33-e08e-4cb9-9227-74911a409720"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:08.014094 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.014049    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-istio-data" (OuterVolumeSpecName: "istio-data") pod "cfd46a33-e08e-4cb9-9227-74911a409720" (UID: "cfd46a33-e08e-4cb9-9227-74911a409720"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:08.015586 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.015562    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cfd46a33-e08e-4cb9-9227-74911a409720" (UID: "cfd46a33-e08e-4cb9-9227-74911a409720"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:08.015693 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.015624    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfd46a33-e08e-4cb9-9227-74911a409720-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cfd46a33-e08e-4cb9-9227-74911a409720" (UID: "cfd46a33-e08e-4cb9-9227-74911a409720"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:45:08.015805 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.015742    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd46a33-e08e-4cb9-9227-74911a409720-kube-api-access-tjkq4" (OuterVolumeSpecName: "kube-api-access-tjkq4") pod "cfd46a33-e08e-4cb9-9227-74911a409720" (UID: "cfd46a33-e08e-4cb9-9227-74911a409720"). InnerVolumeSpecName "kube-api-access-tjkq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:45:08.015805 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.015766    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd46a33-e08e-4cb9-9227-74911a409720-istio-token" (OuterVolumeSpecName: "istio-token") pod "cfd46a33-e08e-4cb9-9227-74911a409720" (UID: "cfd46a33-e08e-4cb9-9227-74911a409720"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:45:08.114456 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.114416    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:08.114456 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.114451    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:08.114456 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.114465    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd46a33-e08e-4cb9-9227-74911a409720-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:08.114704 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.114476    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd46a33-e08e-4cb9-9227-74911a409720-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:08.114704 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.114489    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjkq4\" (UniqueName: \"kubernetes.io/projected/cfd46a33-e08e-4cb9-9227-74911a409720-kube-api-access-tjkq4\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:08.114704 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.114504    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd46a33-e08e-4cb9-9227-74911a409720-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:08.148514 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.148486    2584 generic.go:356] "Generic (PLEG): container finished" podID="cfd46a33-e08e-4cb9-9227-74911a409720" containerID="897cc53f68a9e0d2462ac2ee86cc3c331eea63ee7c973fd8c539becb74ef1a92" exitCode=0
Jul 29 21:45:08.148660 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.148536    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"
Jul 29 21:45:08.148660 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.148556    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" event={"ID":"cfd46a33-e08e-4cb9-9227-74911a409720","Type":"ContainerDied","Data":"897cc53f68a9e0d2462ac2ee86cc3c331eea63ee7c973fd8c539becb74ef1a92"}
Jul 29 21:45:08.148660 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.148600    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b" event={"ID":"cfd46a33-e08e-4cb9-9227-74911a409720","Type":"ContainerDied","Data":"dae8cbb92cb2b75082c8b39428f99959f4fdb8285f2457e92d9b0836a19f3b17"}
Jul 29 21:45:08.148660 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.148623    2584 scope.go:117] "RemoveContainer" containerID="897cc53f68a9e0d2462ac2ee86cc3c331eea63ee7c973fd8c539becb74ef1a92"
Jul 29 21:45:08.157821 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.157805    2584 scope.go:117] "RemoveContainer" containerID="7272c0ab903b037fbbf4372526c52b110164b0eeead11498a785c3e3bc813228"
Jul 29 21:45:08.165234 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.165218    2584 scope.go:117] "RemoveContainer" containerID="897cc53f68a9e0d2462ac2ee86cc3c331eea63ee7c973fd8c539becb74ef1a92"
Jul 29 21:45:08.165495 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:45:08.165472    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897cc53f68a9e0d2462ac2ee86cc3c331eea63ee7c973fd8c539becb74ef1a92\": container with ID starting with 897cc53f68a9e0d2462ac2ee86cc3c331eea63ee7c973fd8c539becb74ef1a92 not found: ID does not exist" containerID="897cc53f68a9e0d2462ac2ee86cc3c331eea63ee7c973fd8c539becb74ef1a92"
Jul 29 21:45:08.165542 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.165519    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897cc53f68a9e0d2462ac2ee86cc3c331eea63ee7c973fd8c539becb74ef1a92"} err="failed to get container status \"897cc53f68a9e0d2462ac2ee86cc3c331eea63ee7c973fd8c539becb74ef1a92\": rpc error: code = NotFound desc = could not find container \"897cc53f68a9e0d2462ac2ee86cc3c331eea63ee7c973fd8c539becb74ef1a92\": container with ID starting with 897cc53f68a9e0d2462ac2ee86cc3c331eea63ee7c973fd8c539becb74ef1a92 not found: ID does not exist"
Jul 29 21:45:08.165542 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.165536    2584 scope.go:117] "RemoveContainer" containerID="7272c0ab903b037fbbf4372526c52b110164b0eeead11498a785c3e3bc813228"
Jul 29 21:45:08.165767 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:45:08.165747    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7272c0ab903b037fbbf4372526c52b110164b0eeead11498a785c3e3bc813228\": container with ID starting with 7272c0ab903b037fbbf4372526c52b110164b0eeead11498a785c3e3bc813228 not found: ID does not exist" containerID="7272c0ab903b037fbbf4372526c52b110164b0eeead11498a785c3e3bc813228"
Jul 29 21:45:08.165809 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.165773    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7272c0ab903b037fbbf4372526c52b110164b0eeead11498a785c3e3bc813228"} err="failed to get container status \"7272c0ab903b037fbbf4372526c52b110164b0eeead11498a785c3e3bc813228\": rpc error: code = NotFound desc = could not find container \"7272c0ab903b037fbbf4372526c52b110164b0eeead11498a785c3e3bc813228\": container with ID starting with 7272c0ab903b037fbbf4372526c52b110164b0eeead11498a785c3e3bc813228 not found: ID does not exist"
Jul 29 21:45:08.169707 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.169686    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"]
Jul 29 21:45:08.173507 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.173488    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bdcc5a3e-openshift-default-7766f7884-t7m9b"]
Jul 29 21:45:08.865267 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:08.865237    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" path="/var/lib/kubelet/pods/cfd46a33-e08e-4cb9-9227-74911a409720/volumes"
Jul 29 21:45:18.495583 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:18.495540    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn"]
Jul 29 21:45:18.500177 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:18.500138    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-641c4002-29756025-gcpjn"]
Jul 29 21:45:18.864835 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:18.864806    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98fb8976-8f8e-48ef-b682-d41f621a29af" path="/var/lib/kubelet/pods/98fb8976-8f8e-48ef-b682-d41f621a29af/volumes"
Jul 29 21:45:25.242619 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:25.242589    2584 scope.go:117] "RemoveContainer" containerID="dcb378cb597ee8b82c31aa6e6ff1082dbfc71b4dfb0c866668ccdd6b7d829e05"
Jul 29 21:45:25.353383 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:25.353345    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"]
Jul 29 21:45:25.353653 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:25.353604    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m" podUID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerName="istio-proxy" containerID="cri-o://deabcdab819f193ca0d35d272786ef1fc731f81934632339225cd1d0c8408118" gracePeriod=30
Jul 29 21:45:30.248697 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.248657    2584 patch_prober.go:28] interesting pod/e2e-mt-641c4002-openshift-default-76f64b9856-b548m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 21:45:30.249084 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.248712    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m" podUID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:45:30.605244 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.605220    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:45:30.795669 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.795587    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-data\") pod \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") "
Jul 29 21:45:30.795669 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.795621    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkzvm\" (UniqueName: \"kubernetes.io/projected/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-kube-api-access-bkzvm\") pod \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") "
Jul 29 21:45:30.795669 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.795655    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-podinfo\") pod \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") "
Jul 29 21:45:30.795940 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.795694    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-workload-certs\") pod \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") "
Jul 29 21:45:30.795940 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.795719    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istiod-ca-cert\") pod \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") "
Jul 29 21:45:30.795940 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.795743    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-envoy\") pod \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") "
Jul 29 21:45:30.795940 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.795865    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-credential-socket\") pod \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") "
Jul 29 21:45:30.795940 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.795910    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-token\") pod \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") "
Jul 29 21:45:30.796249 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.795970    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-workload-socket\") pod \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\" (UID: \"2abbfd11-fd4a-422c-9889-2ea42d4fb06e\") "
Jul 29 21:45:30.796249 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.796064    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2abbfd11-fd4a-422c-9889-2ea42d4fb06e" (UID: "2abbfd11-fd4a-422c-9889-2ea42d4fb06e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:30.796378 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.796226    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2abbfd11-fd4a-422c-9889-2ea42d4fb06e" (UID: "2abbfd11-fd4a-422c-9889-2ea42d4fb06e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:30.796378 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.796284    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2abbfd11-fd4a-422c-9889-2ea42d4fb06e" (UID: "2abbfd11-fd4a-422c-9889-2ea42d4fb06e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:45:30.796378 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.796290    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2abbfd11-fd4a-422c-9889-2ea42d4fb06e" (UID: "2abbfd11-fd4a-422c-9889-2ea42d4fb06e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:30.796378 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.796334    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:30.796378 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.796353    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:30.796731 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.796706    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-data" (OuterVolumeSpecName: "istio-data") pod "2abbfd11-fd4a-422c-9889-2ea42d4fb06e" (UID: "2abbfd11-fd4a-422c-9889-2ea42d4fb06e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:30.798030 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.798001    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2abbfd11-fd4a-422c-9889-2ea42d4fb06e" (UID: "2abbfd11-fd4a-422c-9889-2ea42d4fb06e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:45:30.798220 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.798187    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-kube-api-access-bkzvm" (OuterVolumeSpecName: "kube-api-access-bkzvm") pod "2abbfd11-fd4a-422c-9889-2ea42d4fb06e" (UID: "2abbfd11-fd4a-422c-9889-2ea42d4fb06e"). InnerVolumeSpecName "kube-api-access-bkzvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:45:30.798313 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.798252    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2abbfd11-fd4a-422c-9889-2ea42d4fb06e" (UID: "2abbfd11-fd4a-422c-9889-2ea42d4fb06e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:30.798313 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.798268    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-token" (OuterVolumeSpecName: "istio-token") pod "2abbfd11-fd4a-422c-9889-2ea42d4fb06e" (UID: "2abbfd11-fd4a-422c-9889-2ea42d4fb06e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:45:30.896960 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.896931    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:30.896960 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.896955    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkzvm\" (UniqueName: \"kubernetes.io/projected/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-kube-api-access-bkzvm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:30.896960 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.896965    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:30.897181 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.896973    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:30.897181 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.896981    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:30.897181 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.896989    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:30.897181 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:30.896997    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abbfd11-fd4a-422c-9889-2ea42d4fb06e-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:45:31.252613 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:31.252571    2584 generic.go:356] "Generic (PLEG): container finished" podID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerID="deabcdab819f193ca0d35d272786ef1fc731f81934632339225cd1d0c8408118" exitCode=0
Jul 29 21:45:31.253114 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:31.252618    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m" event={"ID":"2abbfd11-fd4a-422c-9889-2ea42d4fb06e","Type":"ContainerDied","Data":"deabcdab819f193ca0d35d272786ef1fc731f81934632339225cd1d0c8408118"}
Jul 29 21:45:31.253114 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:31.252661    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m" event={"ID":"2abbfd11-fd4a-422c-9889-2ea42d4fb06e","Type":"ContainerDied","Data":"0e89aafed738fcd241174efeed5e63769404e7321ff1c4d8af5cca2d79d2665f"}
Jul 29 21:45:31.253114 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:31.252687    2584 scope.go:117] "RemoveContainer" containerID="deabcdab819f193ca0d35d272786ef1fc731f81934632339225cd1d0c8408118"
Jul 29 21:45:31.253114 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:31.252624    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"
Jul 29 21:45:31.261112 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:31.261096    2584 scope.go:117] "RemoveContainer" containerID="f239fcd181e92b870454e95e96dfd85eb6fe603465bb34309ebea70893e37c5b"
Jul 29 21:45:31.268638 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:31.268622    2584 scope.go:117] "RemoveContainer" containerID="deabcdab819f193ca0d35d272786ef1fc731f81934632339225cd1d0c8408118"
Jul 29 21:45:31.268887 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:45:31.268868    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deabcdab819f193ca0d35d272786ef1fc731f81934632339225cd1d0c8408118\": container with ID starting with deabcdab819f193ca0d35d272786ef1fc731f81934632339225cd1d0c8408118 not found: ID does not exist" containerID="deabcdab819f193ca0d35d272786ef1fc731f81934632339225cd1d0c8408118"
Jul 29 21:45:31.268959 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:31.268893    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deabcdab819f193ca0d35d272786ef1fc731f81934632339225cd1d0c8408118"} err="failed to get container status \"deabcdab819f193ca0d35d272786ef1fc731f81934632339225cd1d0c8408118\": rpc error: code = NotFound desc = could not find container \"deabcdab819f193ca0d35d272786ef1fc731f81934632339225cd1d0c8408118\": container with ID starting with deabcdab819f193ca0d35d272786ef1fc731f81934632339225cd1d0c8408118 not found: ID does not exist"
Jul 29 21:45:31.268959 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:31.268910    2584 scope.go:117] "RemoveContainer" containerID="f239fcd181e92b870454e95e96dfd85eb6fe603465bb34309ebea70893e37c5b"
Jul 29 21:45:31.269189 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:45:31.269164    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f239fcd181e92b870454e95e96dfd85eb6fe603465bb34309ebea70893e37c5b\": container with ID starting with f239fcd181e92b870454e95e96dfd85eb6fe603465bb34309ebea70893e37c5b not found: ID does not exist" containerID="f239fcd181e92b870454e95e96dfd85eb6fe603465bb34309ebea70893e37c5b"
Jul 29 21:45:31.269296 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:31.269191    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f239fcd181e92b870454e95e96dfd85eb6fe603465bb34309ebea70893e37c5b"} err="failed to get container status \"f239fcd181e92b870454e95e96dfd85eb6fe603465bb34309ebea70893e37c5b\": rpc error: code = NotFound desc = could not find container \"f239fcd181e92b870454e95e96dfd85eb6fe603465bb34309ebea70893e37c5b\": container with ID starting with f239fcd181e92b870454e95e96dfd85eb6fe603465bb34309ebea70893e37c5b not found: ID does not exist"
Jul 29 21:45:31.270122 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:31.270096    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"]
Jul 29 21:45:31.273657 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:31.273636    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-641c4002-openshift-default-76f64b9856-b548m"]
Jul 29 21:45:32.865446 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:45:32.865417    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" path="/var/lib/kubelet/pods/2abbfd11-fd4a-422c-9889-2ea42d4fb06e/volumes"
Jul 29 21:46:04.562718 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.562671    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"]
Jul 29 21:46:04.563456 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563346    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerName="istio-proxy"
Jul 29 21:46:04.563456 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563367    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerName="istio-proxy"
Jul 29 21:46:04.563456 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563382    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerName="istio-proxy"
Jul 29 21:46:04.563456 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563391    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerName="istio-proxy"
Jul 29 21:46:04.563456 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563410    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy"
Jul 29 21:46:04.563456 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563419    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy"
Jul 29 21:46:04.563456 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563444    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy"
Jul 29 21:46:04.563456 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563452    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy"
Jul 29 21:46:04.563456 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563462    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f453691-7fea-4373-86b0-316dd0c09139" containerName="cleanup"
Jul 29 21:46:04.563887 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563470    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f453691-7fea-4373-86b0-316dd0c09139" containerName="cleanup"
Jul 29 21:46:04.563887 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563489    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98fb8976-8f8e-48ef-b682-d41f621a29af" containerName="cleanup"
Jul 29 21:46:04.563887 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563497    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="98fb8976-8f8e-48ef-b682-d41f621a29af" containerName="cleanup"
Jul 29 21:46:04.563887 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563639    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy"
Jul 29 21:46:04.563887 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563655    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerName="istio-proxy"
Jul 29 21:46:04.563887 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563665    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abbfd11-fd4a-422c-9889-2ea42d4fb06e" containerName="istio-proxy"
Jul 29 21:46:04.563887 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563696    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f453691-7fea-4373-86b0-316dd0c09139" containerName="cleanup"
Jul 29 21:46:04.563887 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.563742    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="98fb8976-8f8e-48ef-b682-d41f621a29af" containerName="cleanup"
Jul 29 21:46:04.566976 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.566958    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.568917 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.568899    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b0cb4c-openshift-default-dockercfg-hkvc5\""
Jul 29 21:46:04.574997 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.574976    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"]
Jul 29 21:46:04.664830 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.664800    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-workload-certs\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.665019 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.664847    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-credential-socket\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.665019 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.664869    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istiod-ca-cert\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.665019 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.664995    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-data\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.665173 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.665035    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-podinfo\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.665173 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.665097    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-token\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.665173 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.665122    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-workload-socket\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.665297 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.665191    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dftww\" (UniqueName: \"kubernetes.io/projected/ae23e099-3470-4d5a-9b71-0ac2d29b8584-kube-api-access-dftww\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.665297 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.665239    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-envoy\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.766111 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.766071    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-data\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.766111 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.766119    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-podinfo\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.766406 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.766173    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-token\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.766406 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.766193    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-workload-socket\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.766406 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.766235    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftww\" (UniqueName: \"kubernetes.io/projected/ae23e099-3470-4d5a-9b71-0ac2d29b8584-kube-api-access-dftww\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.766406 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.766278    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-envoy\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.766406 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.766309    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-workload-certs\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.766406 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.766346    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-credential-socket\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.766406 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.766371    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istiod-ca-cert\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.766768 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.766559    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-data\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.766768 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.766667    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-workload-socket\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.766768 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.766746    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-workload-certs\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.766937 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.766807    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-credential-socket\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.767255 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.767239    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istiod-ca-cert\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.768722 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.768699    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-envoy\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.768820 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.768773    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-podinfo\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.773981 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.773958    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-token\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.774069 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.774030    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dftww\" (UniqueName: \"kubernetes.io/projected/ae23e099-3470-4d5a-9b71-0ac2d29b8584-kube-api-access-dftww\") pod \"e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") " pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:04.878907 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:04.878878    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:05.010318 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:05.010291    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"]
Jul 29 21:46:05.012664 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:46:05.012635    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae23e099_3470_4d5a_9b71_0ac2d29b8584.slice/crio-2261fa227c049ad7a6ce59ac9425c9c2a30333f0f4fe0c2a6f8343f8fabe61e9 WatchSource:0}: Error finding container 2261fa227c049ad7a6ce59ac9425c9c2a30333f0f4fe0c2a6f8343f8fabe61e9: Status 404 returned error can't find the container with id 2261fa227c049ad7a6ce59ac9425c9c2a30333f0f4fe0c2a6f8343f8fabe61e9
Jul 29 21:46:05.015184 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:05.015099    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:46:05.015269 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:05.015195    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:46:05.015269 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:05.015227    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:46:05.410887 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:05.410854    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7" event={"ID":"ae23e099-3470-4d5a-9b71-0ac2d29b8584","Type":"ContainerStarted","Data":"14538abdef9e7becf2af5f31759e723da7b864817a26724a3a3f11fae497717d"}
Jul 29 21:46:05.411052 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:05.410906    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7" event={"ID":"ae23e099-3470-4d5a-9b71-0ac2d29b8584","Type":"ContainerStarted","Data":"2261fa227c049ad7a6ce59ac9425c9c2a30333f0f4fe0c2a6f8343f8fabe61e9"}
Jul 29 21:46:05.431622 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:05.431568    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7" podStartSLOduration=1.431549189 podStartE2EDuration="1.431549189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:46:05.428065265 +0000 UTC m=+3581.024356949" watchObservedRunningTime="2026-07-29 21:46:05.431549189 +0000 UTC m=+3581.027840875"
Jul 29 21:46:05.878975 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:05.878929    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:06.883742 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:06.883713    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:07.420492 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:07.420462    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:07.421441 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:07.421421    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:21.484227 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:21.484195    2584 generic.go:356] "Generic (PLEG): container finished" podID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" containerID="14538abdef9e7becf2af5f31759e723da7b864817a26724a3a3f11fae497717d" exitCode=0
Jul 29 21:46:21.484632 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:21.484255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7" event={"ID":"ae23e099-3470-4d5a-9b71-0ac2d29b8584","Type":"ContainerDied","Data":"14538abdef9e7becf2af5f31759e723da7b864817a26724a3a3f11fae497717d"}
Jul 29 21:46:21.484632 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:21.484621    2584 scope.go:117] "RemoveContainer" containerID="14538abdef9e7becf2af5f31759e723da7b864817a26724a3a3f11fae497717d"
Jul 29 21:46:21.485525 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:21.485496    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:46:21.485575 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:21.485565    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:46:21.485633 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:21.485612    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:46:22.421262 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:22.421205    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:22.491969 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:22.491921    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7" event={"ID":"ae23e099-3470-4d5a-9b71-0ac2d29b8584","Type":"ContainerStarted","Data":"e519ea3c772514842b65ed0edd77947cb1df9772a61657535ea15ac313de73b8"}
Jul 29 21:46:22.879803 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:22.879769    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:22.880855 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:22.880831    2584 patch_prober.go:28] interesting pod/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 29 21:46:22.880958 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:22.880878    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7" podUID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 29 21:46:23.879536 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:23.879505    2584 patch_prober.go:28] interesting pod/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 29 21:46:23.879932 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:23.879572    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7" podUID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 29 21:46:24.879271 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:24.879240    2584 patch_prober.go:28] interesting pod/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 29 21:46:24.879446 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:24.879296    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7" podUID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 29 21:46:25.264649 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:25.264567    2584 scope.go:117] "RemoveContainer" containerID="8df35cdb16e305a3031d2363912714c5e56e2c067def3afe34cfea3b15e2c83d"
Jul 29 21:46:25.879471 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:25.879426    2584 patch_prober.go:28] interesting pod/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 29 21:46:25.879652 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:25.879503    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7" podUID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 29 21:46:26.882903 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:26.882873    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:26.883302 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:26.883191    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:26.883823 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:26.883805    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:46:33.001291 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.001208    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"]
Jul 29 21:46:33.001823 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.001804    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd46a33-e08e-4cb9-9227-74911a409720" containerName="istio-proxy"
Jul 29 21:46:33.006735 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.006714    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.008957 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.008933    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ceb7ca-openshift-default-dockercfg-zckbl\""
Jul 29 21:46:33.017774 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.017750    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"]
Jul 29 21:46:33.108003 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.107965    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-istio-envoy\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.108206 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.108056    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84398c5f-7f33-4254-8680-9f4a0d231995-istio-token\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.108206 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.108117    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-credential-socket\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.108206 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.108159    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-workload-certs\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.108392 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.108213    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-istio-data\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.108392 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.108256    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84398c5f-7f33-4254-8680-9f4a0d231995-istiod-ca-cert\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.108392 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.108294    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqspx\" (UniqueName: \"kubernetes.io/projected/84398c5f-7f33-4254-8680-9f4a0d231995-kube-api-access-sqspx\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.108392 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.108326    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-workload-socket\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.108392 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.108349    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84398c5f-7f33-4254-8680-9f4a0d231995-istio-podinfo\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.208893 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.208861    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-istio-data\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.209054 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.208902    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84398c5f-7f33-4254-8680-9f4a0d231995-istiod-ca-cert\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.209054 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.208936    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqspx\" (UniqueName: \"kubernetes.io/projected/84398c5f-7f33-4254-8680-9f4a0d231995-kube-api-access-sqspx\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.209054 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.208962    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-workload-socket\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.209054 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.208986    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84398c5f-7f33-4254-8680-9f4a0d231995-istio-podinfo\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.209054 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.209035    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-istio-envoy\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.209368 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.209058    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84398c5f-7f33-4254-8680-9f4a0d231995-istio-token\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.209368 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.209109    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-credential-socket\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.209368 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.209135    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-workload-certs\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.209368 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.209331    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-istio-data\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.209563 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.209379    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-workload-socket\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.209605 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.209557    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-workload-certs\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.209605 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.209592    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-credential-socket\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.209760 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.209739    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84398c5f-7f33-4254-8680-9f4a0d231995-istiod-ca-cert\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.211334 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.211310    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-istio-envoy\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.211567 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.211547    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84398c5f-7f33-4254-8680-9f4a0d231995-istio-podinfo\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.223706 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.223679    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84398c5f-7f33-4254-8680-9f4a0d231995-istio-token\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.225095 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.225073    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqspx\" (UniqueName: \"kubernetes.io/projected/84398c5f-7f33-4254-8680-9f4a0d231995-kube-api-access-sqspx\") pod \"e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") " pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.317202 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.317108    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:33.451619 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.451593    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"]
Jul 29 21:46:33.453808 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:46:33.453783    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84398c5f_7f33_4254_8680_9f4a0d231995.slice/crio-051fa0a90b4d542855fe5bc411f1d213c093fd4118d7c743b2010dfd726e5082 WatchSource:0}: Error finding container 051fa0a90b4d542855fe5bc411f1d213c093fd4118d7c743b2010dfd726e5082: Status 404 returned error can't find the container with id 051fa0a90b4d542855fe5bc411f1d213c093fd4118d7c743b2010dfd726e5082
Jul 29 21:46:33.455887 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.455859    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:46:33.455965 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.455926    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:46:33.455965 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.455956    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:46:33.553042 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.553006    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99" event={"ID":"84398c5f-7f33-4254-8680-9f4a0d231995","Type":"ContainerStarted","Data":"334078bf4d06dd8e302b2e5d506424f80abd5470476ba33742bf67b1a0a3b60e"}
Jul 29 21:46:33.553217 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.553071    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99" event={"ID":"84398c5f-7f33-4254-8680-9f4a0d231995","Type":"ContainerStarted","Data":"051fa0a90b4d542855fe5bc411f1d213c093fd4118d7c743b2010dfd726e5082"}
Jul 29 21:46:33.570852 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:33.570760    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99" podStartSLOduration=1.570742116 podStartE2EDuration="1.570742116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:46:33.570085721 +0000 UTC m=+3609.166377434" watchObservedRunningTime="2026-07-29 21:46:33.570742116 +0000 UTC m=+3609.167033801"
Jul 29 21:46:34.317839 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:34.317803    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:34.322633 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:34.322607    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:34.558284 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:34.558252    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:34.559078 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:34.559063    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:50.585730 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:50.584973    2584 patch_prober.go:28] interesting pod/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 29 21:46:50.585730 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:50.585070    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99" podUID="84398c5f-7f33-4254-8680-9f4a0d231995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 21:46:50.636028 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:50.635996    2584 generic.go:356] "Generic (PLEG): container finished" podID="84398c5f-7f33-4254-8680-9f4a0d231995" containerID="334078bf4d06dd8e302b2e5d506424f80abd5470476ba33742bf67b1a0a3b60e" exitCode=0
Jul 29 21:46:50.636199 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:50.636075    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99" event={"ID":"84398c5f-7f33-4254-8680-9f4a0d231995","Type":"ContainerDied","Data":"334078bf4d06dd8e302b2e5d506424f80abd5470476ba33742bf67b1a0a3b60e"}
Jul 29 21:46:50.636490 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:50.636476    2584 scope.go:117] "RemoveContainer" containerID="334078bf4d06dd8e302b2e5d506424f80abd5470476ba33742bf67b1a0a3b60e"
Jul 29 21:46:50.637450 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:50.637426    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:46:50.637509 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:50.637489    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:46:50.637545 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:50.637522    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:46:51.642526 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:51.642492    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99" event={"ID":"84398c5f-7f33-4254-8680-9f4a0d231995","Type":"ContainerStarted","Data":"2df622da0a3937b54ddb280e0ca46cde9ce11e91e8c6333d42fd44962d5ec36f"}
Jul 29 21:46:52.317471 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:52.317439    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:52.317780 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:52.317756    2584 patch_prober.go:28] interesting pod/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 29 21:46:52.317883 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:52.317811    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99" podUID="84398c5f-7f33-4254-8680-9f4a0d231995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 29 21:46:53.317581 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:53.317541    2584 patch_prober.go:28] interesting pod/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 29 21:46:53.317990 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:53.317596    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99" podUID="84398c5f-7f33-4254-8680-9f4a0d231995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 29 21:46:54.317743 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:54.317699    2584 patch_prober.go:28] interesting pod/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 29 21:46:54.318180 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:54.317772    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99" podUID="84398c5f-7f33-4254-8680-9f4a0d231995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 29 21:46:55.318099 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:55.318059    2584 patch_prober.go:28] interesting pod/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 29 21:46:55.318528 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:55.318136    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99" podUID="84398c5f-7f33-4254-8680-9f4a0d231995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 29 21:46:56.321070 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:56.321037    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:56.321487 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:56.321395    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:46:56.322219 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:46:56.322200    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:47:06.342181 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.342135    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"]
Jul 29 21:47:06.345548 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.345530    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.348417 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.348391    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b0cb4c\"/\"test-model-b0cb4c-kserve-self-signed-certs\""
Jul 29 21:47:06.348417 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.348407    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b0cb4c\"/\"kube-root-ca.crt\""
Jul 29 21:47:06.348616 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.348457    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b0cb4c\"/\"default-dockercfg-62wkf\""
Jul 29 21:47:06.348616 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.348512    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b0cb4c\"/\"openshift-service-ca.crt\""
Jul 29 21:47:06.354739 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.354708    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"]
Jul 29 21:47:06.398998 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.398959    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-model-cache\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.398998 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.399002    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-home\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.399253 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.399088    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-tmp-dir\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.399253 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.399134    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30dc110b-e3ae-4602-b475-1c51281d4cbd-tls-certs\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.399253 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.399193    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-dshm\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.399253 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.399250    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s2kp\" (UniqueName: \"kubernetes.io/projected/30dc110b-e3ae-4602-b475-1c51281d4cbd-kube-api-access-9s2kp\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.499998 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.499962    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-tmp-dir\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.500213 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.500005    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30dc110b-e3ae-4602-b475-1c51281d4cbd-tls-certs\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.500213 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.500037    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-dshm\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.500213 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.500080    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s2kp\" (UniqueName: \"kubernetes.io/projected/30dc110b-e3ae-4602-b475-1c51281d4cbd-kube-api-access-9s2kp\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.500213 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.500104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-model-cache\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.500213 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.500136    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-home\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.500531 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.500512    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-model-cache\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.500613 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.500592    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-home\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.500665 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.500635    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-tmp-dir\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.502355 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.502330    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-dshm\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.502589 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.502573    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30dc110b-e3ae-4602-b475-1c51281d4cbd-tls-certs\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.507654 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.507629    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s2kp\" (UniqueName: \"kubernetes.io/projected/30dc110b-e3ae-4602-b475-1c51281d4cbd-kube-api-access-9s2kp\") pod \"test-model-b0cb4c-kserve-77758b8c57-mmmbr\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") " pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.657535 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.657446    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:06.796139 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:06.796113    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"]
Jul 29 21:47:06.798503 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:47:06.798476    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30dc110b_e3ae_4602_b475_1c51281d4cbd.slice/crio-8709ef942fc2a2f61ac3a8381c875c3fc6702ced37a7dc3e4495bced634a9258 WatchSource:0}: Error finding container 8709ef942fc2a2f61ac3a8381c875c3fc6702ced37a7dc3e4495bced634a9258: Status 404 returned error can't find the container with id 8709ef942fc2a2f61ac3a8381c875c3fc6702ced37a7dc3e4495bced634a9258
Jul 29 21:47:07.718842 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:07.718805    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr" event={"ID":"30dc110b-e3ae-4602-b475-1c51281d4cbd","Type":"ContainerStarted","Data":"4c78338e5356f2f958a3fd12c3a2e3038efcbf659138a955e440000e71f0844a"}
Jul 29 21:47:07.718842 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:07.718849    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr" event={"ID":"30dc110b-e3ae-4602-b475-1c51281d4cbd","Type":"ContainerStarted","Data":"8709ef942fc2a2f61ac3a8381c875c3fc6702ced37a7dc3e4495bced634a9258"}
Jul 29 21:47:16.658254 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:16.658212    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:16.658683 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:16.658274    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:16.671033 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:16.671008    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:16.688295 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:16.688235    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr" podStartSLOduration=10.688214811 podStartE2EDuration="10.688214811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:47:07.736511185 +0000 UTC m=+3643.332802870" watchObservedRunningTime="2026-07-29 21:47:16.688214811 +0000 UTC m=+3652.284506512"
Jul 29 21:47:16.772860 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:16.772827    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:47:27.448756 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.448719    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"]
Jul 29 21:47:27.452643 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.452621    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.455365 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.455343    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ceb7ca\"/\"openshift-service-ca.crt\""
Jul 29 21:47:27.455495 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.455405    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ceb7ca\"/\"default-dockercfg-8c2cd\""
Jul 29 21:47:27.455495 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.455386    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ceb7ca\"/\"test-model-ceb7ca-kserve-self-signed-certs\""
Jul 29 21:47:27.455495 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.455487    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ceb7ca\"/\"kube-root-ca.crt\""
Jul 29 21:47:27.463262 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.463237    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"]
Jul 29 21:47:27.587695 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.587651    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-home\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.587881 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.587716    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7962905e-486a-4608-85fe-8c92ba59c00a-tls-certs\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.587881 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.587821    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxb4l\" (UniqueName: \"kubernetes.io/projected/7962905e-486a-4608-85fe-8c92ba59c00a-kube-api-access-cxb4l\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.587881 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.587857    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-dshm\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.588037 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.587915    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-model-cache\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.588037 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.587958    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-tmp-dir\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.689229 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.689191    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7962905e-486a-4608-85fe-8c92ba59c00a-tls-certs\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.689418 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.689285    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxb4l\" (UniqueName: \"kubernetes.io/projected/7962905e-486a-4608-85fe-8c92ba59c00a-kube-api-access-cxb4l\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.689418 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.689321    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-dshm\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.689418 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.689358    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-model-cache\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.689418 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.689386    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-tmp-dir\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.689650 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.689446    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-home\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.689793 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.689767    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-model-cache\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.689861 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.689805    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-tmp-dir\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.689910 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.689865    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-home\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.691567 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.691546    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-dshm\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.691740 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.691721    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7962905e-486a-4608-85fe-8c92ba59c00a-tls-certs\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.697375 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.697354    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxb4l\" (UniqueName: \"kubernetes.io/projected/7962905e-486a-4608-85fe-8c92ba59c00a-kube-api-access-cxb4l\") pod \"test-model-ceb7ca-kserve-8559f966d7-lmfzp\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") " pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.762459 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.762367    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:27.889926 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:27.889900    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"]
Jul 29 21:47:27.891911 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:47:27.891886    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7962905e_486a_4608_85fe_8c92ba59c00a.slice/crio-84f3853deac27ad37fce145ddc7d42cee2a8375c17d4565e6def2f5b1d64d180 WatchSource:0}: Error finding container 84f3853deac27ad37fce145ddc7d42cee2a8375c17d4565e6def2f5b1d64d180: Status 404 returned error can't find the container with id 84f3853deac27ad37fce145ddc7d42cee2a8375c17d4565e6def2f5b1d64d180
Jul 29 21:47:28.822026 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:28.821986    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp" event={"ID":"7962905e-486a-4608-85fe-8c92ba59c00a","Type":"ContainerStarted","Data":"9230bbe18ba8439bf51812346e6228334d72835aae69f3c2a1d9486c4a4e829a"}
Jul 29 21:47:28.822026 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:28.822031    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp" event={"ID":"7962905e-486a-4608-85fe-8c92ba59c00a","Type":"ContainerStarted","Data":"84f3853deac27ad37fce145ddc7d42cee2a8375c17d4565e6def2f5b1d64d180"}
Jul 29 21:47:28.839812 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:28.839761    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp" podStartSLOduration=1.839744689 podStartE2EDuration="1.839744689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:47:28.837708865 +0000 UTC m=+3664.434000548" watchObservedRunningTime="2026-07-29 21:47:28.839744689 +0000 UTC m=+3664.436036372"
Jul 29 21:47:37.763493 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:37.763449    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:37.763896 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:37.763516    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:37.776320 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:37.776291    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:47:37.874249 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:47:37.874215    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:48:15.012332 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:15.012301    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"]
Jul 29 21:48:15.975665 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:15.975630    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"]
Jul 29 21:48:30.031304 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.031237    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr" podUID="30dc110b-e3ae-4602-b475-1c51281d4cbd" containerName="main" containerID="cri-o://4c78338e5356f2f958a3fd12c3a2e3038efcbf659138a955e440000e71f0844a" gracePeriod=45
Jul 29 21:48:30.273576 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.273555    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:48:30.341749 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.341726    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s2kp\" (UniqueName: \"kubernetes.io/projected/30dc110b-e3ae-4602-b475-1c51281d4cbd-kube-api-access-9s2kp\") pod \"30dc110b-e3ae-4602-b475-1c51281d4cbd\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") "
Jul 29 21:48:30.341890 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.341807    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-home\") pod \"30dc110b-e3ae-4602-b475-1c51281d4cbd\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") "
Jul 29 21:48:30.341890 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.341822    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30dc110b-e3ae-4602-b475-1c51281d4cbd-tls-certs\") pod \"30dc110b-e3ae-4602-b475-1c51281d4cbd\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") "
Jul 29 21:48:30.341890 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.341850    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-tmp-dir\") pod \"30dc110b-e3ae-4602-b475-1c51281d4cbd\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") "
Jul 29 21:48:30.341890 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.341879    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-model-cache\") pod \"30dc110b-e3ae-4602-b475-1c51281d4cbd\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") "
Jul 29 21:48:30.342067 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.341973    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-dshm\") pod \"30dc110b-e3ae-4602-b475-1c51281d4cbd\" (UID: \"30dc110b-e3ae-4602-b475-1c51281d4cbd\") "
Jul 29 21:48:30.342133 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.342115    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-home" (OuterVolumeSpecName: "home") pod "30dc110b-e3ae-4602-b475-1c51281d4cbd" (UID: "30dc110b-e3ae-4602-b475-1c51281d4cbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:30.342227 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.342131    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-model-cache" (OuterVolumeSpecName: "model-cache") pod "30dc110b-e3ae-4602-b475-1c51281d4cbd" (UID: "30dc110b-e3ae-4602-b475-1c51281d4cbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:30.342227 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.342170    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30dc110b-e3ae-4602-b475-1c51281d4cbd" (UID: "30dc110b-e3ae-4602-b475-1c51281d4cbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:30.342349 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.342337    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:30.342397 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.342356    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:30.342397 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.342377    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:30.344032 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.344008    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-dshm" (OuterVolumeSpecName: "dshm") pod "30dc110b-e3ae-4602-b475-1c51281d4cbd" (UID: "30dc110b-e3ae-4602-b475-1c51281d4cbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:30.344183 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.344055    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30dc110b-e3ae-4602-b475-1c51281d4cbd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30dc110b-e3ae-4602-b475-1c51281d4cbd" (UID: "30dc110b-e3ae-4602-b475-1c51281d4cbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:48:30.344183 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.344058    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30dc110b-e3ae-4602-b475-1c51281d4cbd-kube-api-access-9s2kp" (OuterVolumeSpecName: "kube-api-access-9s2kp") pod "30dc110b-e3ae-4602-b475-1c51281d4cbd" (UID: "30dc110b-e3ae-4602-b475-1c51281d4cbd"). InnerVolumeSpecName "kube-api-access-9s2kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:48:30.443438 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.443403    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30dc110b-e3ae-4602-b475-1c51281d4cbd-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:30.443438 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.443433    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30dc110b-e3ae-4602-b475-1c51281d4cbd-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:30.443438 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.443443    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s2kp\" (UniqueName: \"kubernetes.io/projected/30dc110b-e3ae-4602-b475-1c51281d4cbd-kube-api-access-9s2kp\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:30.995291 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:30.995250    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp" podUID="7962905e-486a-4608-85fe-8c92ba59c00a" containerName="main" containerID="cri-o://9230bbe18ba8439bf51812346e6228334d72835aae69f3c2a1d9486c4a4e829a" gracePeriod=45
Jul 29 21:48:31.105881 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.105843    2584 generic.go:356] "Generic (PLEG): container finished" podID="30dc110b-e3ae-4602-b475-1c51281d4cbd" containerID="4c78338e5356f2f958a3fd12c3a2e3038efcbf659138a955e440000e71f0844a" exitCode=0
Jul 29 21:48:31.106330 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.105891    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"
Jul 29 21:48:31.106330 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.105895    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr" event={"ID":"30dc110b-e3ae-4602-b475-1c51281d4cbd","Type":"ContainerDied","Data":"4c78338e5356f2f958a3fd12c3a2e3038efcbf659138a955e440000e71f0844a"}
Jul 29 21:48:31.106330 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.105936    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr" event={"ID":"30dc110b-e3ae-4602-b475-1c51281d4cbd","Type":"ContainerDied","Data":"8709ef942fc2a2f61ac3a8381c875c3fc6702ced37a7dc3e4495bced634a9258"}
Jul 29 21:48:31.106330 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.105958    2584 scope.go:117] "RemoveContainer" containerID="4c78338e5356f2f958a3fd12c3a2e3038efcbf659138a955e440000e71f0844a"
Jul 29 21:48:31.114625 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.114602    2584 scope.go:117] "RemoveContainer" containerID="4c78338e5356f2f958a3fd12c3a2e3038efcbf659138a955e440000e71f0844a"
Jul 29 21:48:31.114882 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:48:31.114861    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c78338e5356f2f958a3fd12c3a2e3038efcbf659138a955e440000e71f0844a\": container with ID starting with 4c78338e5356f2f958a3fd12c3a2e3038efcbf659138a955e440000e71f0844a not found: ID does not exist" containerID="4c78338e5356f2f958a3fd12c3a2e3038efcbf659138a955e440000e71f0844a"
Jul 29 21:48:31.114939 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.114891    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c78338e5356f2f958a3fd12c3a2e3038efcbf659138a955e440000e71f0844a"} err="failed to get container status \"4c78338e5356f2f958a3fd12c3a2e3038efcbf659138a955e440000e71f0844a\": rpc error: code = NotFound desc = could not find container \"4c78338e5356f2f958a3fd12c3a2e3038efcbf659138a955e440000e71f0844a\": container with ID starting with 4c78338e5356f2f958a3fd12c3a2e3038efcbf659138a955e440000e71f0844a not found: ID does not exist"
Jul 29 21:48:31.123173 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.123093    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"]
Jul 29 21:48:31.125727 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.125704    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b0cb4c/test-model-b0cb4c-kserve-77758b8c57-mmmbr"]
Jul 29 21:48:31.257579 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.257550    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:48:31.351422 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.351376    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7962905e-486a-4608-85fe-8c92ba59c00a-tls-certs\") pod \"7962905e-486a-4608-85fe-8c92ba59c00a\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") "
Jul 29 21:48:31.351422 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.351427    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-model-cache\") pod \"7962905e-486a-4608-85fe-8c92ba59c00a\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") "
Jul 29 21:48:31.351674 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.351448    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-home\") pod \"7962905e-486a-4608-85fe-8c92ba59c00a\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") "
Jul 29 21:48:31.351674 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.351474    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-dshm\") pod \"7962905e-486a-4608-85fe-8c92ba59c00a\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") "
Jul 29 21:48:31.351674 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.351490    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-tmp-dir\") pod \"7962905e-486a-4608-85fe-8c92ba59c00a\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") "
Jul 29 21:48:31.351674 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.351571    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxb4l\" (UniqueName: \"kubernetes.io/projected/7962905e-486a-4608-85fe-8c92ba59c00a-kube-api-access-cxb4l\") pod \"7962905e-486a-4608-85fe-8c92ba59c00a\" (UID: \"7962905e-486a-4608-85fe-8c92ba59c00a\") "
Jul 29 21:48:31.351885 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.351705    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7962905e-486a-4608-85fe-8c92ba59c00a" (UID: "7962905e-486a-4608-85fe-8c92ba59c00a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:31.351885 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.351754    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-home" (OuterVolumeSpecName: "home") pod "7962905e-486a-4608-85fe-8c92ba59c00a" (UID: "7962905e-486a-4608-85fe-8c92ba59c00a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:31.351885 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.351774    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7962905e-486a-4608-85fe-8c92ba59c00a" (UID: "7962905e-486a-4608-85fe-8c92ba59c00a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:31.352017 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.351896    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:31.352017 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.351914    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:31.352017 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.351928    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:31.353804 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.353778    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7962905e-486a-4608-85fe-8c92ba59c00a-kube-api-access-cxb4l" (OuterVolumeSpecName: "kube-api-access-cxb4l") pod "7962905e-486a-4608-85fe-8c92ba59c00a" (UID: "7962905e-486a-4608-85fe-8c92ba59c00a"). InnerVolumeSpecName "kube-api-access-cxb4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:48:31.353913 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.353819    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7962905e-486a-4608-85fe-8c92ba59c00a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7962905e-486a-4608-85fe-8c92ba59c00a" (UID: "7962905e-486a-4608-85fe-8c92ba59c00a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:48:31.353913 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.353819    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-dshm" (OuterVolumeSpecName: "dshm") pod "7962905e-486a-4608-85fe-8c92ba59c00a" (UID: "7962905e-486a-4608-85fe-8c92ba59c00a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:31.452629 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.452532    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxb4l\" (UniqueName: \"kubernetes.io/projected/7962905e-486a-4608-85fe-8c92ba59c00a-kube-api-access-cxb4l\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:31.452629 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.452567    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7962905e-486a-4608-85fe-8c92ba59c00a-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:31.452629 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:31.452580    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7962905e-486a-4608-85fe-8c92ba59c00a-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:32.111535 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:32.111503    2584 generic.go:356] "Generic (PLEG): container finished" podID="7962905e-486a-4608-85fe-8c92ba59c00a" containerID="9230bbe18ba8439bf51812346e6228334d72835aae69f3c2a1d9486c4a4e829a" exitCode=0
Jul 29 21:48:32.111997 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:32.111561    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"
Jul 29 21:48:32.111997 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:32.111591    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp" event={"ID":"7962905e-486a-4608-85fe-8c92ba59c00a","Type":"ContainerDied","Data":"9230bbe18ba8439bf51812346e6228334d72835aae69f3c2a1d9486c4a4e829a"}
Jul 29 21:48:32.111997 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:32.111640    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp" event={"ID":"7962905e-486a-4608-85fe-8c92ba59c00a","Type":"ContainerDied","Data":"84f3853deac27ad37fce145ddc7d42cee2a8375c17d4565e6def2f5b1d64d180"}
Jul 29 21:48:32.111997 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:32.111666    2584 scope.go:117] "RemoveContainer" containerID="9230bbe18ba8439bf51812346e6228334d72835aae69f3c2a1d9486c4a4e829a"
Jul 29 21:48:32.122672 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:32.122654    2584 scope.go:117] "RemoveContainer" containerID="9230bbe18ba8439bf51812346e6228334d72835aae69f3c2a1d9486c4a4e829a"
Jul 29 21:48:32.122941 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:48:32.122921    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9230bbe18ba8439bf51812346e6228334d72835aae69f3c2a1d9486c4a4e829a\": container with ID starting with 9230bbe18ba8439bf51812346e6228334d72835aae69f3c2a1d9486c4a4e829a not found: ID does not exist" containerID="9230bbe18ba8439bf51812346e6228334d72835aae69f3c2a1d9486c4a4e829a"
Jul 29 21:48:32.123035 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:32.122946    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9230bbe18ba8439bf51812346e6228334d72835aae69f3c2a1d9486c4a4e829a"} err="failed to get container status \"9230bbe18ba8439bf51812346e6228334d72835aae69f3c2a1d9486c4a4e829a\": rpc error: code = NotFound desc = could not find container \"9230bbe18ba8439bf51812346e6228334d72835aae69f3c2a1d9486c4a4e829a\": container with ID starting with 9230bbe18ba8439bf51812346e6228334d72835aae69f3c2a1d9486c4a4e829a not found: ID does not exist"
Jul 29 21:48:32.143619 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:32.143589    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"]
Jul 29 21:48:32.147427 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:32.147405    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ceb7ca/test-model-ceb7ca-kserve-8559f966d7-lmfzp"]
Jul 29 21:48:32.865767 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:32.865732    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30dc110b-e3ae-4602-b475-1c51281d4cbd" path="/var/lib/kubelet/pods/30dc110b-e3ae-4602-b475-1c51281d4cbd/volumes"
Jul 29 21:48:32.866179 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:32.866164    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7962905e-486a-4608-85fe-8c92ba59c00a" path="/var/lib/kubelet/pods/7962905e-486a-4608-85fe-8c92ba59c00a/volumes"
Jul 29 21:48:33.257083 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:33.256999    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"]
Jul 29 21:48:33.257502 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:33.257294    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7" podUID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" containerName="istio-proxy" containerID="cri-o://e519ea3c772514842b65ed0edd77947cb1df9772a61657535ea15ac313de73b8" gracePeriod=30
Jul 29 21:48:38.516045 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.516022    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:48:38.623889 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.623854    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-envoy\") pod \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") "
Jul 29 21:48:38.623889 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.623893    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-workload-certs\") pod \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") "
Jul 29 21:48:38.624179 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.623918    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-workload-socket\") pod \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") "
Jul 29 21:48:38.624179 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.623947    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-token\") pod \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") "
Jul 29 21:48:38.624179 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.623973    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-podinfo\") pod \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") "
Jul 29 21:48:38.624179 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.624000    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-credential-socket\") pod \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") "
Jul 29 21:48:38.624179 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.624062    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istiod-ca-cert\") pod \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") "
Jul 29 21:48:38.624461 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.624202    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-data\") pod \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") "
Jul 29 21:48:38.624461 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.624256    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dftww\" (UniqueName: \"kubernetes.io/projected/ae23e099-3470-4d5a-9b71-0ac2d29b8584-kube-api-access-dftww\") pod \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\" (UID: \"ae23e099-3470-4d5a-9b71-0ac2d29b8584\") "
Jul 29 21:48:38.624461 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.624206    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ae23e099-3470-4d5a-9b71-0ac2d29b8584" (UID: "ae23e099-3470-4d5a-9b71-0ac2d29b8584"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:38.624461 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.624271    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ae23e099-3470-4d5a-9b71-0ac2d29b8584" (UID: "ae23e099-3470-4d5a-9b71-0ac2d29b8584"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:38.624690 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.624542    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:38.624690 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.624561    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:38.624690 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.624544    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ae23e099-3470-4d5a-9b71-0ac2d29b8584" (UID: "ae23e099-3470-4d5a-9b71-0ac2d29b8584"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:38.624848 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.624805    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ae23e099-3470-4d5a-9b71-0ac2d29b8584" (UID: "ae23e099-3470-4d5a-9b71-0ac2d29b8584"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:48:38.625163 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.625121    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-data" (OuterVolumeSpecName: "istio-data") pod "ae23e099-3470-4d5a-9b71-0ac2d29b8584" (UID: "ae23e099-3470-4d5a-9b71-0ac2d29b8584"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:38.626352 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.626325    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ae23e099-3470-4d5a-9b71-0ac2d29b8584" (UID: "ae23e099-3470-4d5a-9b71-0ac2d29b8584"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:38.626464 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.626400    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ae23e099-3470-4d5a-9b71-0ac2d29b8584" (UID: "ae23e099-3470-4d5a-9b71-0ac2d29b8584"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:48:38.626464 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.626420    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae23e099-3470-4d5a-9b71-0ac2d29b8584-kube-api-access-dftww" (OuterVolumeSpecName: "kube-api-access-dftww") pod "ae23e099-3470-4d5a-9b71-0ac2d29b8584" (UID: "ae23e099-3470-4d5a-9b71-0ac2d29b8584"). InnerVolumeSpecName "kube-api-access-dftww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:48:38.626464 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.626409    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-token" (OuterVolumeSpecName: "istio-token") pod "ae23e099-3470-4d5a-9b71-0ac2d29b8584" (UID: "ae23e099-3470-4d5a-9b71-0ac2d29b8584"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:48:38.725636 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.725595    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:38.725636 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.725633    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:38.725636 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.725644    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:38.725869 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.725652    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:38.725869 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.725662    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:38.725869 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.725671    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae23e099-3470-4d5a-9b71-0ac2d29b8584-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:38.725869 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:38.725678    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dftww\" (UniqueName: \"kubernetes.io/projected/ae23e099-3470-4d5a-9b71-0ac2d29b8584-kube-api-access-dftww\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:39.145916 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:39.145881    2584 generic.go:356] "Generic (PLEG): container finished" podID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" containerID="e519ea3c772514842b65ed0edd77947cb1df9772a61657535ea15ac313de73b8" exitCode=0
Jul 29 21:48:39.146111 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:39.145944    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"
Jul 29 21:48:39.146111 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:39.145955    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7" event={"ID":"ae23e099-3470-4d5a-9b71-0ac2d29b8584","Type":"ContainerDied","Data":"e519ea3c772514842b65ed0edd77947cb1df9772a61657535ea15ac313de73b8"}
Jul 29 21:48:39.146111 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:39.146006    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7" event={"ID":"ae23e099-3470-4d5a-9b71-0ac2d29b8584","Type":"ContainerDied","Data":"2261fa227c049ad7a6ce59ac9425c9c2a30333f0f4fe0c2a6f8343f8fabe61e9"}
Jul 29 21:48:39.146111 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:39.146028    2584 scope.go:117] "RemoveContainer" containerID="e519ea3c772514842b65ed0edd77947cb1df9772a61657535ea15ac313de73b8"
Jul 29 21:48:39.154836 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:39.154813    2584 scope.go:117] "RemoveContainer" containerID="14538abdef9e7becf2af5f31759e723da7b864817a26724a3a3f11fae497717d"
Jul 29 21:48:39.162628 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:39.162610    2584 scope.go:117] "RemoveContainer" containerID="e519ea3c772514842b65ed0edd77947cb1df9772a61657535ea15ac313de73b8"
Jul 29 21:48:39.162906 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:48:39.162886    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e519ea3c772514842b65ed0edd77947cb1df9772a61657535ea15ac313de73b8\": container with ID starting with e519ea3c772514842b65ed0edd77947cb1df9772a61657535ea15ac313de73b8 not found: ID does not exist" containerID="e519ea3c772514842b65ed0edd77947cb1df9772a61657535ea15ac313de73b8"
Jul 29 21:48:39.162959 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:39.162914    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e519ea3c772514842b65ed0edd77947cb1df9772a61657535ea15ac313de73b8"} err="failed to get container status \"e519ea3c772514842b65ed0edd77947cb1df9772a61657535ea15ac313de73b8\": rpc error: code = NotFound desc = could not find container \"e519ea3c772514842b65ed0edd77947cb1df9772a61657535ea15ac313de73b8\": container with ID starting with e519ea3c772514842b65ed0edd77947cb1df9772a61657535ea15ac313de73b8 not found: ID does not exist"
Jul 29 21:48:39.162959 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:39.162932    2584 scope.go:117] "RemoveContainer" containerID="14538abdef9e7becf2af5f31759e723da7b864817a26724a3a3f11fae497717d"
Jul 29 21:48:39.163204 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:48:39.163187    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14538abdef9e7becf2af5f31759e723da7b864817a26724a3a3f11fae497717d\": container with ID starting with 14538abdef9e7becf2af5f31759e723da7b864817a26724a3a3f11fae497717d not found: ID does not exist" containerID="14538abdef9e7becf2af5f31759e723da7b864817a26724a3a3f11fae497717d"
Jul 29 21:48:39.163256 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:39.163215    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14538abdef9e7becf2af5f31759e723da7b864817a26724a3a3f11fae497717d"} err="failed to get container status \"14538abdef9e7becf2af5f31759e723da7b864817a26724a3a3f11fae497717d\": rpc error: code = NotFound desc = could not find container \"14538abdef9e7becf2af5f31759e723da7b864817a26724a3a3f11fae497717d\": container with ID starting with 14538abdef9e7becf2af5f31759e723da7b864817a26724a3a3f11fae497717d not found: ID does not exist"
Jul 29 21:48:39.163957 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:39.163940    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"]
Jul 29 21:48:39.170663 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:39.170643    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b0cb4c-openshift-default-655fbbbb5d-brnk7"]
Jul 29 21:48:40.865062 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:40.865026    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" path="/var/lib/kubelet/pods/ae23e099-3470-4d5a-9b71-0ac2d29b8584/volumes"
Jul 29 21:48:50.749101 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:50.749065    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"]
Jul 29 21:48:50.749660 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:50.749313    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99" podUID="84398c5f-7f33-4254-8680-9f4a0d231995" containerName="istio-proxy" containerID="cri-o://2df622da0a3937b54ddb280e0ca46cde9ce11e91e8c6333d42fd44962d5ec36f" gracePeriod=30
Jul 29 21:48:51.663051 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.663014    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"]
Jul 29 21:48:51.663358 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.663346    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7962905e-486a-4608-85fe-8c92ba59c00a" containerName="main"
Jul 29 21:48:51.663408 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.663361    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7962905e-486a-4608-85fe-8c92ba59c00a" containerName="main"
Jul 29 21:48:51.663408 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.663373    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" containerName="istio-proxy"
Jul 29 21:48:51.663408 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.663378    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" containerName="istio-proxy"
Jul 29 21:48:51.663408 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.663386    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30dc110b-e3ae-4602-b475-1c51281d4cbd" containerName="main"
Jul 29 21:48:51.663408 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.663391    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="30dc110b-e3ae-4602-b475-1c51281d4cbd" containerName="main"
Jul 29 21:48:51.663565 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.663453    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" containerName="istio-proxy"
Jul 29 21:48:51.663565 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.663460    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="30dc110b-e3ae-4602-b475-1c51281d4cbd" containerName="main"
Jul 29 21:48:51.663565 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.663466    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" containerName="istio-proxy"
Jul 29 21:48:51.663565 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.663472    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7962905e-486a-4608-85fe-8c92ba59c00a" containerName="main"
Jul 29 21:48:51.666584 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.666553    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.669724 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.669704    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fdb37a-openshift-default-dockercfg-26xqj\""
Jul 29 21:48:51.685524 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.685496    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"]
Jul 29 21:48:51.734631 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.734595    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-podinfo\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.734631 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.734633    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-token\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.734830 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.734665    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-workload-socket\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.734830 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.734682    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-credential-socket\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.734830 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.734766    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-workload-certs\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.734830 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.734798    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-data\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.734830 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.734815    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv9z2\" (UniqueName: \"kubernetes.io/projected/5db92c96-14af-413b-b86b-428f3dc3d2dd-kube-api-access-mv9z2\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.735014 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.734878    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db92c96-14af-413b-b86b-428f3dc3d2dd-istiod-ca-cert\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.735014 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.734900    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-envoy\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.835687 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.835656    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-envoy\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.836196 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.835706    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-podinfo\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.836196 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.835726    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-token\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.836196 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.835751    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-workload-socket\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.836196 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.835767    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-credential-socket\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.836196 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.835803    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-workload-certs\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.836196 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.835820    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-data\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.836196 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.835835    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv9z2\" (UniqueName: \"kubernetes.io/projected/5db92c96-14af-413b-b86b-428f3dc3d2dd-kube-api-access-mv9z2\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.836196 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.835860    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db92c96-14af-413b-b86b-428f3dc3d2dd-istiod-ca-cert\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.836196 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.836167    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-workload-socket\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.836707 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.836271    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-credential-socket\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.836707 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.836314    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-data\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.836707 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.836380    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-workload-certs\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.836892 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.836725    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db92c96-14af-413b-b86b-428f3dc3d2dd-istiod-ca-cert\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.838117 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.838095    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-envoy\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.838381 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.838363    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-podinfo\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.843561 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.843544    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-token\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.843838 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.843817    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv9z2\" (UniqueName: \"kubernetes.io/projected/5db92c96-14af-413b-b86b-428f3dc3d2dd-kube-api-access-mv9z2\") pod \"e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") " pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:51.976959 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:51.976873    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:52.112176 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:52.112131    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"]
Jul 29 21:48:52.114442 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:48:52.114415    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db92c96_14af_413b_b86b_428f3dc3d2dd.slice/crio-413377175e6d3cd0f541cb3ed5c8c758f5f833f2acca7c32570f8e463c943fd6 WatchSource:0}: Error finding container 413377175e6d3cd0f541cb3ed5c8c758f5f833f2acca7c32570f8e463c943fd6: Status 404 returned error can't find the container with id 413377175e6d3cd0f541cb3ed5c8c758f5f833f2acca7c32570f8e463c943fd6
Jul 29 21:48:52.116577 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:52.116546    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:48:52.116686 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:52.116622    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:48:52.116686 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:52.116664    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:48:52.208373 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:52.208337    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22" event={"ID":"5db92c96-14af-413b-b86b-428f3dc3d2dd","Type":"ContainerStarted","Data":"afb1a1eee73a48a1bf80694fed736e235add47ff29bcdab0dbe9f259be5cacce"}
Jul 29 21:48:52.208510 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:52.208403    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22" event={"ID":"5db92c96-14af-413b-b86b-428f3dc3d2dd","Type":"ContainerStarted","Data":"413377175e6d3cd0f541cb3ed5c8c758f5f833f2acca7c32570f8e463c943fd6"}
Jul 29 21:48:52.231797 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:52.231693    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22" podStartSLOduration=1.231670949 podStartE2EDuration="1.231670949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:48:52.228923773 +0000 UTC m=+3747.825215476" watchObservedRunningTime="2026-07-29 21:48:52.231670949 +0000 UTC m=+3747.827962633"
Jul 29 21:48:52.977263 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:52.977212    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:53.982808 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:53.982774    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:54.217400 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:54.217366    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:54.218322 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:54.218302    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:48:56.006766 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.006743    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:48:56.175187 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175135    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-istio-data\") pod \"84398c5f-7f33-4254-8680-9f4a0d231995\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") "
Jul 29 21:48:56.175372 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175195    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-workload-socket\") pod \"84398c5f-7f33-4254-8680-9f4a0d231995\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") "
Jul 29 21:48:56.175372 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175226    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84398c5f-7f33-4254-8680-9f4a0d231995-istio-podinfo\") pod \"84398c5f-7f33-4254-8680-9f4a0d231995\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") "
Jul 29 21:48:56.175372 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175242    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-istio-envoy\") pod \"84398c5f-7f33-4254-8680-9f4a0d231995\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") "
Jul 29 21:48:56.175372 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175288    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84398c5f-7f33-4254-8680-9f4a0d231995-istiod-ca-cert\") pod \"84398c5f-7f33-4254-8680-9f4a0d231995\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") "
Jul 29 21:48:56.175372 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175308    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-credential-socket\") pod \"84398c5f-7f33-4254-8680-9f4a0d231995\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") "
Jul 29 21:48:56.175372 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175325    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqspx\" (UniqueName: \"kubernetes.io/projected/84398c5f-7f33-4254-8680-9f4a0d231995-kube-api-access-sqspx\") pod \"84398c5f-7f33-4254-8680-9f4a0d231995\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") "
Jul 29 21:48:56.175372 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175355    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84398c5f-7f33-4254-8680-9f4a0d231995-istio-token\") pod \"84398c5f-7f33-4254-8680-9f4a0d231995\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") "
Jul 29 21:48:56.175742 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175413    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-workload-certs\") pod \"84398c5f-7f33-4254-8680-9f4a0d231995\" (UID: \"84398c5f-7f33-4254-8680-9f4a0d231995\") "
Jul 29 21:48:56.175742 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175485    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "84398c5f-7f33-4254-8680-9f4a0d231995" (UID: "84398c5f-7f33-4254-8680-9f4a0d231995"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:56.175742 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175631    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "84398c5f-7f33-4254-8680-9f4a0d231995" (UID: "84398c5f-7f33-4254-8680-9f4a0d231995"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:56.175742 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175665    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:56.175742 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175729    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84398c5f-7f33-4254-8680-9f4a0d231995-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "84398c5f-7f33-4254-8680-9f4a0d231995" (UID: "84398c5f-7f33-4254-8680-9f4a0d231995"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:48:56.176011 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.175802    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "84398c5f-7f33-4254-8680-9f4a0d231995" (UID: "84398c5f-7f33-4254-8680-9f4a0d231995"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:56.176106 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.176082    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-istio-data" (OuterVolumeSpecName: "istio-data") pod "84398c5f-7f33-4254-8680-9f4a0d231995" (UID: "84398c5f-7f33-4254-8680-9f4a0d231995"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:56.177582 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.177553    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "84398c5f-7f33-4254-8680-9f4a0d231995" (UID: "84398c5f-7f33-4254-8680-9f4a0d231995"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:48:56.177706 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.177667    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84398c5f-7f33-4254-8680-9f4a0d231995-istio-token" (OuterVolumeSpecName: "istio-token") pod "84398c5f-7f33-4254-8680-9f4a0d231995" (UID: "84398c5f-7f33-4254-8680-9f4a0d231995"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:48:56.177706 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.177684    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84398c5f-7f33-4254-8680-9f4a0d231995-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "84398c5f-7f33-4254-8680-9f4a0d231995" (UID: "84398c5f-7f33-4254-8680-9f4a0d231995"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:48:56.177802 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.177690    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84398c5f-7f33-4254-8680-9f4a0d231995-kube-api-access-sqspx" (OuterVolumeSpecName: "kube-api-access-sqspx") pod "84398c5f-7f33-4254-8680-9f4a0d231995" (UID: "84398c5f-7f33-4254-8680-9f4a0d231995"). InnerVolumeSpecName "kube-api-access-sqspx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:48:56.229326 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.229294    2584 generic.go:356] "Generic (PLEG): container finished" podID="84398c5f-7f33-4254-8680-9f4a0d231995" containerID="2df622da0a3937b54ddb280e0ca46cde9ce11e91e8c6333d42fd44962d5ec36f" exitCode=0
Jul 29 21:48:56.229487 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.229344    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"
Jul 29 21:48:56.229487 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.229385    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99" event={"ID":"84398c5f-7f33-4254-8680-9f4a0d231995","Type":"ContainerDied","Data":"2df622da0a3937b54ddb280e0ca46cde9ce11e91e8c6333d42fd44962d5ec36f"}
Jul 29 21:48:56.229487 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.229438    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99" event={"ID":"84398c5f-7f33-4254-8680-9f4a0d231995","Type":"ContainerDied","Data":"051fa0a90b4d542855fe5bc411f1d213c093fd4118d7c743b2010dfd726e5082"}
Jul 29 21:48:56.229487 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.229462    2584 scope.go:117] "RemoveContainer" containerID="2df622da0a3937b54ddb280e0ca46cde9ce11e91e8c6333d42fd44962d5ec36f"
Jul 29 21:48:56.239906 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.239886    2584 scope.go:117] "RemoveContainer" containerID="334078bf4d06dd8e302b2e5d506424f80abd5470476ba33742bf67b1a0a3b60e"
Jul 29 21:48:56.247472 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.247455    2584 scope.go:117] "RemoveContainer" containerID="2df622da0a3937b54ddb280e0ca46cde9ce11e91e8c6333d42fd44962d5ec36f"
Jul 29 21:48:56.247748 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:48:56.247730    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df622da0a3937b54ddb280e0ca46cde9ce11e91e8c6333d42fd44962d5ec36f\": container with ID starting with 2df622da0a3937b54ddb280e0ca46cde9ce11e91e8c6333d42fd44962d5ec36f not found: ID does not exist" containerID="2df622da0a3937b54ddb280e0ca46cde9ce11e91e8c6333d42fd44962d5ec36f"
Jul 29 21:48:56.247787 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.247757    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df622da0a3937b54ddb280e0ca46cde9ce11e91e8c6333d42fd44962d5ec36f"} err="failed to get container status \"2df622da0a3937b54ddb280e0ca46cde9ce11e91e8c6333d42fd44962d5ec36f\": rpc error: code = NotFound desc = could not find container \"2df622da0a3937b54ddb280e0ca46cde9ce11e91e8c6333d42fd44962d5ec36f\": container with ID starting with 2df622da0a3937b54ddb280e0ca46cde9ce11e91e8c6333d42fd44962d5ec36f not found: ID does not exist"
Jul 29 21:48:56.247787 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.247774    2584 scope.go:117] "RemoveContainer" containerID="334078bf4d06dd8e302b2e5d506424f80abd5470476ba33742bf67b1a0a3b60e"
Jul 29 21:48:56.248008 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:48:56.247991    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334078bf4d06dd8e302b2e5d506424f80abd5470476ba33742bf67b1a0a3b60e\": container with ID starting with 334078bf4d06dd8e302b2e5d506424f80abd5470476ba33742bf67b1a0a3b60e not found: ID does not exist" containerID="334078bf4d06dd8e302b2e5d506424f80abd5470476ba33742bf67b1a0a3b60e"
Jul 29 21:48:56.248042 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.248015    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334078bf4d06dd8e302b2e5d506424f80abd5470476ba33742bf67b1a0a3b60e"} err="failed to get container status \"334078bf4d06dd8e302b2e5d506424f80abd5470476ba33742bf67b1a0a3b60e\": rpc error: code = NotFound desc = could not find container \"334078bf4d06dd8e302b2e5d506424f80abd5470476ba33742bf67b1a0a3b60e\": container with ID starting with 334078bf4d06dd8e302b2e5d506424f80abd5470476ba33742bf67b1a0a3b60e not found: ID does not exist"
Jul 29 21:48:56.251818 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.251796    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"]
Jul 29 21:48:56.254793 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.254774    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ceb7ca-openshift-default-6689bcff4b-jxt99"]
Jul 29 21:48:56.276317 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.276293    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:56.276317 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.276314    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84398c5f-7f33-4254-8680-9f4a0d231995-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:56.276452 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.276324    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:56.276452 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.276335    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84398c5f-7f33-4254-8680-9f4a0d231995-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:56.276452 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.276343    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:56.276452 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.276352    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqspx\" (UniqueName: \"kubernetes.io/projected/84398c5f-7f33-4254-8680-9f4a0d231995-kube-api-access-sqspx\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:56.276452 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.276361    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84398c5f-7f33-4254-8680-9f4a0d231995-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:56.276452 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.276369    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84398c5f-7f33-4254-8680-9f4a0d231995-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:48:56.865979 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:48:56.865945    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84398c5f-7f33-4254-8680-9f4a0d231995" path="/var/lib/kubelet/pods/84398c5f-7f33-4254-8680-9f4a0d231995/volumes"
Jul 29 21:49:08.284613 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:08.284578    2584 generic.go:356] "Generic (PLEG): container finished" podID="5db92c96-14af-413b-b86b-428f3dc3d2dd" containerID="afb1a1eee73a48a1bf80694fed736e235add47ff29bcdab0dbe9f259be5cacce" exitCode=0
Jul 29 21:49:08.285029 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:08.284639    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22" event={"ID":"5db92c96-14af-413b-b86b-428f3dc3d2dd","Type":"ContainerDied","Data":"afb1a1eee73a48a1bf80694fed736e235add47ff29bcdab0dbe9f259be5cacce"}
Jul 29 21:49:08.285029 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:08.284981    2584 scope.go:117] "RemoveContainer" containerID="afb1a1eee73a48a1bf80694fed736e235add47ff29bcdab0dbe9f259be5cacce"
Jul 29 21:49:08.285963 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:08.285933    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:08.286021 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:08.286004    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:08.286057 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:08.286038    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:09.218090 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:09.218031    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:49:09.293924 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:09.293886    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22" event={"ID":"5db92c96-14af-413b-b86b-428f3dc3d2dd","Type":"ContainerStarted","Data":"11885f2c60127f2362a7d6ac11cc5d745d6c8bcf2fbc2ac174024f1171ff81ba"}
Jul 29 21:49:09.977783 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:09.977744    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:49:09.978921 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:09.978894    2584 patch_prober.go:28] interesting pod/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 29 21:49:09.979053 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:09.978945    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22" podUID="5db92c96-14af-413b-b86b-428f3dc3d2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 29 21:49:10.978216 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:10.978178    2584 patch_prober.go:28] interesting pod/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 29 21:49:10.978645 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:10.978230    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22" podUID="5db92c96-14af-413b-b86b-428f3dc3d2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 29 21:49:11.977953 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:11.977914    2584 patch_prober.go:28] interesting pod/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 29 21:49:11.978160 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:11.977982    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22" podUID="5db92c96-14af-413b-b86b-428f3dc3d2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 29 21:49:12.977916 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:12.977875    2584 patch_prober.go:28] interesting pod/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 29 21:49:12.978349 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:12.977945    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22" podUID="5db92c96-14af-413b-b86b-428f3dc3d2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 29 21:49:13.981755 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:13.981721    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:49:13.982138 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:13.982023    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:49:13.982747 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:13.982730    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:49:20.185878 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.185799    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"]
Jul 29 21:49:20.186308 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.186118    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" containerName="istio-proxy"
Jul 29 21:49:20.186308 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.186128    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae23e099-3470-4d5a-9b71-0ac2d29b8584" containerName="istio-proxy"
Jul 29 21:49:20.186308 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.186140    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84398c5f-7f33-4254-8680-9f4a0d231995" containerName="istio-proxy"
Jul 29 21:49:20.186308 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.186171    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="84398c5f-7f33-4254-8680-9f4a0d231995" containerName="istio-proxy"
Jul 29 21:49:20.186308 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.186186    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84398c5f-7f33-4254-8680-9f4a0d231995" containerName="istio-proxy"
Jul 29 21:49:20.186308 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.186194    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="84398c5f-7f33-4254-8680-9f4a0d231995" containerName="istio-proxy"
Jul 29 21:49:20.186308 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.186263    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="84398c5f-7f33-4254-8680-9f4a0d231995" containerName="istio-proxy"
Jul 29 21:49:20.189382 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.189360    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.191263 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.191247    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-25008f-openshift-default-dockercfg-bw7h9\""
Jul 29 21:49:20.204492 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.204464    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"]
Jul 29 21:49:20.286048 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.286015    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwk4\" (UniqueName: \"kubernetes.io/projected/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-kube-api-access-lbwk4\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.286254 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.286072    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-envoy\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.286254 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.286103    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-podinfo\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.286254 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.286200    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-token\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.286385 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.286282    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istiod-ca-cert\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.286385 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.286315    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-workload-certs\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.286385 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.286336    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-credential-socket\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.286385 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.286365    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-workload-socket\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.286527 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.286387    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-data\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.387941 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.387902    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istiod-ca-cert\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.388183 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.387960    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-workload-certs\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.388183 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.387998    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-credential-socket\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.388183 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.388033    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-workload-socket\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.388183 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.388055    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-data\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.388183 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.388096    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwk4\" (UniqueName: \"kubernetes.io/projected/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-kube-api-access-lbwk4\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.388183 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.388134    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-envoy\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.388183 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.388173    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-podinfo\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.388585 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.388210    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-token\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.389178 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.389133    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-data\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.389387 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.389204    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-workload-certs\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.389519 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.389315    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istiod-ca-cert\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.389628 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.389466    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-credential-socket\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.389730 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.389678    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-workload-socket\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.392214 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.392189    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-envoy\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.392335 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.392261    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-podinfo\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.395473 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.395444    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-token\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.396199 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.396130    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwk4\" (UniqueName: \"kubernetes.io/projected/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-kube-api-access-lbwk4\") pod \"e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") " pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.499577 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.499507    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:20.628795 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.628770    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"]
Jul 29 21:49:20.630596 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:49:20.630569    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7d37bd_2365_49d5_bf89_23f296b8b1e3.slice/crio-ff723a8ade7d6ec979a50d4768963a4b4df96af842972e63f76af5922b4ee894 WatchSource:0}: Error finding container ff723a8ade7d6ec979a50d4768963a4b4df96af842972e63f76af5922b4ee894: Status 404 returned error can't find the container with id ff723a8ade7d6ec979a50d4768963a4b4df96af842972e63f76af5922b4ee894
Jul 29 21:49:20.632832 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.632798    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:20.632910 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.632876    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:20.632949 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:20.632923    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:21.356764 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:21.356727    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b" event={"ID":"cd7d37bd-2365-49d5-bf89-23f296b8b1e3","Type":"ContainerStarted","Data":"0680b9ff9a2a8f576c56cbddbda880f1288b6473fbbac6e66a8bdd334b5784fa"}
Jul 29 21:49:21.357172 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:21.356780    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b" event={"ID":"cd7d37bd-2365-49d5-bf89-23f296b8b1e3","Type":"ContainerStarted","Data":"ff723a8ade7d6ec979a50d4768963a4b4df96af842972e63f76af5922b4ee894"}
Jul 29 21:49:21.376303 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:21.376252    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b" podStartSLOduration=1.37623578 podStartE2EDuration="1.37623578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:49:21.375078889 +0000 UTC m=+3776.971370586" watchObservedRunningTime="2026-07-29 21:49:21.37623578 +0000 UTC m=+3776.972527464"
Jul 29 21:49:21.499964 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:21.499931    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:21.504830 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:21.504805    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:22.361031 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:22.361002    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:22.362004 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:22.361985    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:37.841261 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:37.839859    2584 patch_prober.go:28] interesting pod/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": read tcp 10.133.0.2:51780->10.133.0.64:15021: read: connection reset by peer" start-of-body=
Jul 29 21:49:37.841261 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:37.839922    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b" podUID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": read tcp 10.133.0.2:51780->10.133.0.64:15021: read: connection reset by peer"
Jul 29 21:49:38.442195 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:38.442135    2584 generic.go:356] "Generic (PLEG): container finished" podID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerID="0680b9ff9a2a8f576c56cbddbda880f1288b6473fbbac6e66a8bdd334b5784fa" exitCode=0
Jul 29 21:49:38.442371 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:38.442211    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b" event={"ID":"cd7d37bd-2365-49d5-bf89-23f296b8b1e3","Type":"ContainerDied","Data":"0680b9ff9a2a8f576c56cbddbda880f1288b6473fbbac6e66a8bdd334b5784fa"}
Jul 29 21:49:38.442583 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:38.442570    2584 scope.go:117] "RemoveContainer" containerID="0680b9ff9a2a8f576c56cbddbda880f1288b6473fbbac6e66a8bdd334b5784fa"
Jul 29 21:49:38.443539 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:38.443512    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:38.443647 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:38.443592    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:38.443647 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:38.443637    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:39.448763 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:39.448732    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b" event={"ID":"cd7d37bd-2365-49d5-bf89-23f296b8b1e3","Type":"ContainerStarted","Data":"e549c7ff2471d68bc7056e81194cc2c4d953418fda599a8828cf622b08189e88"}
Jul 29 21:49:39.500353 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:39.500245    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:39.500638 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:39.500608    2584 patch_prober.go:28] interesting pod/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 29 21:49:39.500842 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:39.500816    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b" podUID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 29 21:49:40.500243 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:40.500204    2584 patch_prober.go:28] interesting pod/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 29 21:49:40.500744 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:40.500273    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b" podUID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 29 21:49:41.500707 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:41.500666    2584 patch_prober.go:28] interesting pod/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 29 21:49:41.501242 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:41.500729    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b" podUID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 29 21:49:42.500120 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:42.500077    2584 patch_prober.go:28] interesting pod/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 29 21:49:42.500347 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:42.500159    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b" podUID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 29 21:49:43.503890 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:43.503858    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:43.504278 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:43.504104    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:43.504726 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:43.504707    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:49:51.316516 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:51.316480    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5"]
Jul 29 21:49:51.317063 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:51.317043    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="84398c5f-7f33-4254-8680-9f4a0d231995" containerName="istio-proxy"
Jul 29 21:49:51.319912 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:51.319888    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5"
Jul 29 21:49:51.322178 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:51.322131    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:49:51.322407 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:51.322164    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\""
Jul 29 21:49:51.328302 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:51.328280    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5"]
Jul 29 21:49:51.453313 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:51.453259    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f62f04-efac-4dc8-83ef-9684204bcc91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5\" (UID: \"04f62f04-efac-4dc8-83ef-9684204bcc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5"
Jul 29 21:49:51.554764 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:51.554728    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f62f04-efac-4dc8-83ef-9684204bcc91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5\" (UID: \"04f62f04-efac-4dc8-83ef-9684204bcc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5"
Jul 29 21:49:51.555425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:51.555406    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f62f04-efac-4dc8-83ef-9684204bcc91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5\" (UID: \"04f62f04-efac-4dc8-83ef-9684204bcc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5"
Jul 29 21:49:51.631203 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:51.631171    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5"
Jul 29 21:49:51.756796 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:51.756768    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5"]
Jul 29 21:49:51.758747 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:49:51.758721    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f62f04_efac_4dc8_83ef_9684204bcc91.slice/crio-6898612a6a30f8ffdfbdbd80e260ed4479861c16151451251718e9494b36e4d4 WatchSource:0}: Error finding container 6898612a6a30f8ffdfbdbd80e260ed4479861c16151451251718e9494b36e4d4: Status 404 returned error can't find the container with id 6898612a6a30f8ffdfbdbd80e260ed4479861c16151451251718e9494b36e4d4
Jul 29 21:49:52.507114 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:52.507077    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5" event={"ID":"04f62f04-efac-4dc8-83ef-9684204bcc91","Type":"ContainerStarted","Data":"cfa8697ff7b5dbca07bc06b49ed1fb9338c755e6dcb954477bd7b340f42c83bf"}
Jul 29 21:49:52.507552 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:52.507137    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5" event={"ID":"04f62f04-efac-4dc8-83ef-9684204bcc91","Type":"ContainerStarted","Data":"6898612a6a30f8ffdfbdbd80e260ed4479861c16151451251718e9494b36e4d4"}
Jul 29 21:49:52.521765 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:52.521720    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5" podStartSLOduration=1.521705233 podStartE2EDuration="1.521705233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:49:52.520276118 +0000 UTC m=+3808.116567814" watchObservedRunningTime="2026-07-29 21:49:52.521705233 +0000 UTC m=+3808.117996916"
Jul 29 21:49:53.513415 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:53.513380    2584 generic.go:356] "Generic (PLEG): container finished" podID="04f62f04-efac-4dc8-83ef-9684204bcc91" containerID="cfa8697ff7b5dbca07bc06b49ed1fb9338c755e6dcb954477bd7b340f42c83bf" exitCode=0
Jul 29 21:49:53.513904 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:53.513463    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5" event={"ID":"04f62f04-efac-4dc8-83ef-9684204bcc91","Type":"ContainerDied","Data":"cfa8697ff7b5dbca07bc06b49ed1fb9338c755e6dcb954477bd7b340f42c83bf"}
Jul 29 21:49:54.645978 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:54.645949    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5"
Jul 29 21:49:54.783519 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:54.783424    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f62f04-efac-4dc8-83ef-9684204bcc91-maas-api-service-ca\") pod \"04f62f04-efac-4dc8-83ef-9684204bcc91\" (UID: \"04f62f04-efac-4dc8-83ef-9684204bcc91\") "
Jul 29 21:49:54.783853 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:54.783820    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f62f04-efac-4dc8-83ef-9684204bcc91-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04f62f04-efac-4dc8-83ef-9684204bcc91" (UID: "04f62f04-efac-4dc8-83ef-9684204bcc91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:49:54.884318 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:54.884293    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f62f04-efac-4dc8-83ef-9684204bcc91-maas-api-service-ca\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:49:55.525104 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:55.525074    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5" event={"ID":"04f62f04-efac-4dc8-83ef-9684204bcc91","Type":"ContainerDied","Data":"6898612a6a30f8ffdfbdbd80e260ed4479861c16151451251718e9494b36e4d4"}
Jul 29 21:49:55.525325 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:55.525126    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6898612a6a30f8ffdfbdbd80e260ed4479861c16151451251718e9494b36e4d4"
Jul 29 21:49:55.525325 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:49:55.525081    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5"
Jul 29 21:50:07.112713 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:07.112680    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5"]
Jul 29 21:50:07.115078 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:07.115054    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fdb37a-hmrl5"]
Jul 29 21:50:08.865488 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:08.865451    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04f62f04-efac-4dc8-83ef-9684204bcc91" path="/var/lib/kubelet/pods/04f62f04-efac-4dc8-83ef-9684204bcc91/volumes"
Jul 29 21:50:14.142558 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:14.142521    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"]
Jul 29 21:50:14.143033 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:14.142739    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22" podUID="5db92c96-14af-413b-b86b-428f3dc3d2dd" containerName="istio-proxy" containerID="cri-o://11885f2c60127f2362a7d6ac11cc5d745d6c8bcf2fbc2ac174024f1171ff81ba" gracePeriod=30
Jul 29 21:50:19.400907 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.400883    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:50:19.492660 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.492574    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-podinfo\") pod \"5db92c96-14af-413b-b86b-428f3dc3d2dd\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") "
Jul 29 21:50:19.492660 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.492608    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-envoy\") pod \"5db92c96-14af-413b-b86b-428f3dc3d2dd\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") "
Jul 29 21:50:19.492660 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.492633    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-credential-socket\") pod \"5db92c96-14af-413b-b86b-428f3dc3d2dd\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") "
Jul 29 21:50:19.492660 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.492654    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-data\") pod \"5db92c96-14af-413b-b86b-428f3dc3d2dd\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") "
Jul 29 21:50:19.493002 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.492671    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-token\") pod \"5db92c96-14af-413b-b86b-428f3dc3d2dd\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") "
Jul 29 21:50:19.493002 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.492707    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-workload-certs\") pod \"5db92c96-14af-413b-b86b-428f3dc3d2dd\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") "
Jul 29 21:50:19.493002 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.492730    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-workload-socket\") pod \"5db92c96-14af-413b-b86b-428f3dc3d2dd\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") "
Jul 29 21:50:19.493002 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.492777    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db92c96-14af-413b-b86b-428f3dc3d2dd-istiod-ca-cert\") pod \"5db92c96-14af-413b-b86b-428f3dc3d2dd\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") "
Jul 29 21:50:19.493002 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.492831    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv9z2\" (UniqueName: \"kubernetes.io/projected/5db92c96-14af-413b-b86b-428f3dc3d2dd-kube-api-access-mv9z2\") pod \"5db92c96-14af-413b-b86b-428f3dc3d2dd\" (UID: \"5db92c96-14af-413b-b86b-428f3dc3d2dd\") "
Jul 29 21:50:19.493002 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.492972    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5db92c96-14af-413b-b86b-428f3dc3d2dd" (UID: "5db92c96-14af-413b-b86b-428f3dc3d2dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:19.493363 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.493099    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:19.493363 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.493083    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5db92c96-14af-413b-b86b-428f3dc3d2dd" (UID: "5db92c96-14af-413b-b86b-428f3dc3d2dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:19.493363 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.493104    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5db92c96-14af-413b-b86b-428f3dc3d2dd" (UID: "5db92c96-14af-413b-b86b-428f3dc3d2dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:19.493535 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.493401    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db92c96-14af-413b-b86b-428f3dc3d2dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5db92c96-14af-413b-b86b-428f3dc3d2dd" (UID: "5db92c96-14af-413b-b86b-428f3dc3d2dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:50:19.493668 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.493650    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "5db92c96-14af-413b-b86b-428f3dc3d2dd" (UID: "5db92c96-14af-413b-b86b-428f3dc3d2dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:19.495045 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.495014    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5db92c96-14af-413b-b86b-428f3dc3d2dd" (UID: "5db92c96-14af-413b-b86b-428f3dc3d2dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:19.495173 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.495067    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db92c96-14af-413b-b86b-428f3dc3d2dd-kube-api-access-mv9z2" (OuterVolumeSpecName: "kube-api-access-mv9z2") pod "5db92c96-14af-413b-b86b-428f3dc3d2dd" (UID: "5db92c96-14af-413b-b86b-428f3dc3d2dd"). InnerVolumeSpecName "kube-api-access-mv9z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:50:19.495244 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.495223    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5db92c96-14af-413b-b86b-428f3dc3d2dd" (UID: "5db92c96-14af-413b-b86b-428f3dc3d2dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:50:19.495313 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.495268    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "5db92c96-14af-413b-b86b-428f3dc3d2dd" (UID: "5db92c96-14af-413b-b86b-428f3dc3d2dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:50:19.594295 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.594261    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv9z2\" (UniqueName: \"kubernetes.io/projected/5db92c96-14af-413b-b86b-428f3dc3d2dd-kube-api-access-mv9z2\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:19.594295 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.594296    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:19.594504 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.594308    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:19.594504 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.594320    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:19.594504 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.594330    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db92c96-14af-413b-b86b-428f3dc3d2dd-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:19.594504 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.594342    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:19.594504 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.594353    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db92c96-14af-413b-b86b-428f3dc3d2dd-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:19.594504 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.594364    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db92c96-14af-413b-b86b-428f3dc3d2dd-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:19.631461 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.631430    2584 generic.go:356] "Generic (PLEG): container finished" podID="5db92c96-14af-413b-b86b-428f3dc3d2dd" containerID="11885f2c60127f2362a7d6ac11cc5d745d6c8bcf2fbc2ac174024f1171ff81ba" exitCode=0
Jul 29 21:50:19.631588 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.631488    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"
Jul 29 21:50:19.631588 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.631513    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22" event={"ID":"5db92c96-14af-413b-b86b-428f3dc3d2dd","Type":"ContainerDied","Data":"11885f2c60127f2362a7d6ac11cc5d745d6c8bcf2fbc2ac174024f1171ff81ba"}
Jul 29 21:50:19.631680 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.631591    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22" event={"ID":"5db92c96-14af-413b-b86b-428f3dc3d2dd","Type":"ContainerDied","Data":"413377175e6d3cd0f541cb3ed5c8c758f5f833f2acca7c32570f8e463c943fd6"}
Jul 29 21:50:19.631680 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.631617    2584 scope.go:117] "RemoveContainer" containerID="11885f2c60127f2362a7d6ac11cc5d745d6c8bcf2fbc2ac174024f1171ff81ba"
Jul 29 21:50:19.640947 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.640932    2584 scope.go:117] "RemoveContainer" containerID="afb1a1eee73a48a1bf80694fed736e235add47ff29bcdab0dbe9f259be5cacce"
Jul 29 21:50:19.648425 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.648409    2584 scope.go:117] "RemoveContainer" containerID="11885f2c60127f2362a7d6ac11cc5d745d6c8bcf2fbc2ac174024f1171ff81ba"
Jul 29 21:50:19.648684 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:50:19.648668    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11885f2c60127f2362a7d6ac11cc5d745d6c8bcf2fbc2ac174024f1171ff81ba\": container with ID starting with 11885f2c60127f2362a7d6ac11cc5d745d6c8bcf2fbc2ac174024f1171ff81ba not found: ID does not exist" containerID="11885f2c60127f2362a7d6ac11cc5d745d6c8bcf2fbc2ac174024f1171ff81ba"
Jul 29 21:50:19.648740 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.648691    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11885f2c60127f2362a7d6ac11cc5d745d6c8bcf2fbc2ac174024f1171ff81ba"} err="failed to get container status \"11885f2c60127f2362a7d6ac11cc5d745d6c8bcf2fbc2ac174024f1171ff81ba\": rpc error: code = NotFound desc = could not find container \"11885f2c60127f2362a7d6ac11cc5d745d6c8bcf2fbc2ac174024f1171ff81ba\": container with ID starting with 11885f2c60127f2362a7d6ac11cc5d745d6c8bcf2fbc2ac174024f1171ff81ba not found: ID does not exist"
Jul 29 21:50:19.648740 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.648705    2584 scope.go:117] "RemoveContainer" containerID="afb1a1eee73a48a1bf80694fed736e235add47ff29bcdab0dbe9f259be5cacce"
Jul 29 21:50:19.648916 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:50:19.648896    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb1a1eee73a48a1bf80694fed736e235add47ff29bcdab0dbe9f259be5cacce\": container with ID starting with afb1a1eee73a48a1bf80694fed736e235add47ff29bcdab0dbe9f259be5cacce not found: ID does not exist" containerID="afb1a1eee73a48a1bf80694fed736e235add47ff29bcdab0dbe9f259be5cacce"
Jul 29 21:50:19.648976 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.648920    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb1a1eee73a48a1bf80694fed736e235add47ff29bcdab0dbe9f259be5cacce"} err="failed to get container status \"afb1a1eee73a48a1bf80694fed736e235add47ff29bcdab0dbe9f259be5cacce\": rpc error: code = NotFound desc = could not find container \"afb1a1eee73a48a1bf80694fed736e235add47ff29bcdab0dbe9f259be5cacce\": container with ID starting with afb1a1eee73a48a1bf80694fed736e235add47ff29bcdab0dbe9f259be5cacce not found: ID does not exist"
Jul 29 21:50:19.652489 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.652469    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"]
Jul 29 21:50:19.658504 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:19.658483    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fdb37a-openshift-default-bcd7c97db-zjq22"]
Jul 29 21:50:20.864860 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:20.864826    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5db92c96-14af-413b-b86b-428f3dc3d2dd" path="/var/lib/kubelet/pods/5db92c96-14af-413b-b86b-428f3dc3d2dd/volumes"
Jul 29 21:50:36.766723 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:36.766688    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"]
Jul 29 21:50:36.767135 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:36.766933    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b" podUID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerName="istio-proxy" containerID="cri-o://e549c7ff2471d68bc7056e81194cc2c4d953418fda599a8828cf622b08189e88" gracePeriod=30
Jul 29 21:50:37.646027 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.645994    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"]
Jul 29 21:50:37.646354 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.646339    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db92c96-14af-413b-b86b-428f3dc3d2dd" containerName="istio-proxy"
Jul 29 21:50:37.646412 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.646355    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db92c96-14af-413b-b86b-428f3dc3d2dd" containerName="istio-proxy"
Jul 29 21:50:37.646412 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.646363    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04f62f04-efac-4dc8-83ef-9684204bcc91" containerName="revoke-keys"
Jul 29 21:50:37.646412 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.646369    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="04f62f04-efac-4dc8-83ef-9684204bcc91" containerName="revoke-keys"
Jul 29 21:50:37.646518 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.646430    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db92c96-14af-413b-b86b-428f3dc3d2dd" containerName="istio-proxy"
Jul 29 21:50:37.646518 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.646441    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="04f62f04-efac-4dc8-83ef-9684204bcc91" containerName="revoke-keys"
Jul 29 21:50:37.650776 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.650759    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.652675 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.652653    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-b69e52-openshift-default-dockercfg-wdklp\""
Jul 29 21:50:37.664357 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.664328    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"]
Jul 29 21:50:37.744461 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.744430    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvc5\" (UniqueName: \"kubernetes.io/projected/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-kube-api-access-stvc5\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.744651 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.744470    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-workload-socket\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.744651 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.744494    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-podinfo\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.744651 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.744529    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-workload-certs\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.744651 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.744574    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-envoy\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.744651 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.744605    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-credential-socket\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.744651 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.744625    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-data\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.744651 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.744650    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istiod-ca-cert\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.744908 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.744745    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-token\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.845685 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.845654    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istiod-ca-cert\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.846192 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.845698    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-token\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.846192 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.845717    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvc5\" (UniqueName: \"kubernetes.io/projected/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-kube-api-access-stvc5\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.846192 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.845737    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-workload-socket\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.846192 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.845762    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-podinfo\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.846192 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.845783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-workload-certs\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.846192 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.845818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-envoy\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.846192 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.845867    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-credential-socket\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.846192 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.845993    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-data\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.846561 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.846306    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-workload-socket\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.846561 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.846401    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-credential-socket\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.846561 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.846409    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-data\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.846561 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.846521    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istiod-ca-cert\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.846561 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.846527    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-workload-certs\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.848241 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.848214    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-envoy\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.848450 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.848431    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-podinfo\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.852766 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.852742    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-token\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.852922 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.852902    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvc5\" (UniqueName: \"kubernetes.io/projected/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-kube-api-access-stvc5\") pod \"e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") " pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:37.962037 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:37.961955    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:38.091647 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:38.091620    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"]
Jul 29 21:50:38.094110 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:50:38.094082    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfeb8cf_dd51_4bf9_aede_b37f29f4b8ed.slice/crio-30a13618d70bb2f756602dbe95ff2b4092bfbd5811caac39df124de8d635c870 WatchSource:0}: Error finding container 30a13618d70bb2f756602dbe95ff2b4092bfbd5811caac39df124de8d635c870: Status 404 returned error can't find the container with id 30a13618d70bb2f756602dbe95ff2b4092bfbd5811caac39df124de8d635c870
Jul 29 21:50:38.096102 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:38.096071    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:38.096216 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:38.096133    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:38.096216 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:38.096194    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:38.718314 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:38.718280    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt" event={"ID":"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed","Type":"ContainerStarted","Data":"e4802ff0b4bcadf991d3b29b76f0e0312eadc4412f90bdf00cd5614f7245c41c"}
Jul 29 21:50:38.718519 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:38.718332    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt" event={"ID":"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed","Type":"ContainerStarted","Data":"30a13618d70bb2f756602dbe95ff2b4092bfbd5811caac39df124de8d635c870"}
Jul 29 21:50:38.737707 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:38.737655    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt" podStartSLOduration=1.737639994 podStartE2EDuration="1.737639994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:50:38.736057922 +0000 UTC m=+3854.332349628" watchObservedRunningTime="2026-07-29 21:50:38.737639994 +0000 UTC m=+3854.333931751"
Jul 29 21:50:38.962797 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:38.962760    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:39.967777 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:39.967746    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:40.727606 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:40.727578    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:40.728536 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:40.728517    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:42.015840 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.015813    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:50:42.079884 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.079803    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbwk4\" (UniqueName: \"kubernetes.io/projected/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-kube-api-access-lbwk4\") pod \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") "
Jul 29 21:50:42.079884 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.079835    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-credential-socket\") pod \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") "
Jul 29 21:50:42.080110 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.079887    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-token\") pod \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") "
Jul 29 21:50:42.080110 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.079905    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istiod-ca-cert\") pod \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") "
Jul 29 21:50:42.080110 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.079924    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-workload-socket\") pod \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") "
Jul 29 21:50:42.080110 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.079946    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-workload-certs\") pod \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") "
Jul 29 21:50:42.080110 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.079975    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-data\") pod \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") "
Jul 29 21:50:42.080110 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.080043    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-podinfo\") pod \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") "
Jul 29 21:50:42.080110 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.080071    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-envoy\") pod \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\" (UID: \"cd7d37bd-2365-49d5-bf89-23f296b8b1e3\") "
Jul 29 21:50:42.080512 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.080269    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd7d37bd-2365-49d5-bf89-23f296b8b1e3" (UID: "cd7d37bd-2365-49d5-bf89-23f296b8b1e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:42.080512 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.080289    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd7d37bd-2365-49d5-bf89-23f296b8b1e3" (UID: "cd7d37bd-2365-49d5-bf89-23f296b8b1e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:42.080512 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.080392    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd7d37bd-2365-49d5-bf89-23f296b8b1e3" (UID: "cd7d37bd-2365-49d5-bf89-23f296b8b1e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:50:42.080512 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.080403    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:42.080512 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.080423    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:42.080794 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.080640    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd7d37bd-2365-49d5-bf89-23f296b8b1e3" (UID: "cd7d37bd-2365-49d5-bf89-23f296b8b1e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:42.080983 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.080963    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd7d37bd-2365-49d5-bf89-23f296b8b1e3" (UID: "cd7d37bd-2365-49d5-bf89-23f296b8b1e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:42.082168 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.082121    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-kube-api-access-lbwk4" (OuterVolumeSpecName: "kube-api-access-lbwk4") pod "cd7d37bd-2365-49d5-bf89-23f296b8b1e3" (UID: "cd7d37bd-2365-49d5-bf89-23f296b8b1e3"). InnerVolumeSpecName "kube-api-access-lbwk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:50:42.082342 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.082321    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd7d37bd-2365-49d5-bf89-23f296b8b1e3" (UID: "cd7d37bd-2365-49d5-bf89-23f296b8b1e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:50:42.082545 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.082519    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd7d37bd-2365-49d5-bf89-23f296b8b1e3" (UID: "cd7d37bd-2365-49d5-bf89-23f296b8b1e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:50:42.082637 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.082531    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd7d37bd-2365-49d5-bf89-23f296b8b1e3" (UID: "cd7d37bd-2365-49d5-bf89-23f296b8b1e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:42.181087 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.181059    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:42.181087 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.181084    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:42.181087 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.181095    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbwk4\" (UniqueName: \"kubernetes.io/projected/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-kube-api-access-lbwk4\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:42.181334 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.181104    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:42.181334 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.181113    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:42.181334 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.181121    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:42.181334 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.181128    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7d37bd-2365-49d5-bf89-23f296b8b1e3-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:50:42.740516 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.740483    2584 generic.go:356] "Generic (PLEG): container finished" podID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerID="e549c7ff2471d68bc7056e81194cc2c4d953418fda599a8828cf622b08189e88" exitCode=0
Jul 29 21:50:42.740712 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.740537    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"
Jul 29 21:50:42.740712 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.740566    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b" event={"ID":"cd7d37bd-2365-49d5-bf89-23f296b8b1e3","Type":"ContainerDied","Data":"e549c7ff2471d68bc7056e81194cc2c4d953418fda599a8828cf622b08189e88"}
Jul 29 21:50:42.740712 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.740621    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b" event={"ID":"cd7d37bd-2365-49d5-bf89-23f296b8b1e3","Type":"ContainerDied","Data":"ff723a8ade7d6ec979a50d4768963a4b4df96af842972e63f76af5922b4ee894"}
Jul 29 21:50:42.740712 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.740644    2584 scope.go:117] "RemoveContainer" containerID="e549c7ff2471d68bc7056e81194cc2c4d953418fda599a8828cf622b08189e88"
Jul 29 21:50:42.750000 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.749983    2584 scope.go:117] "RemoveContainer" containerID="0680b9ff9a2a8f576c56cbddbda880f1288b6473fbbac6e66a8bdd334b5784fa"
Jul 29 21:50:42.757489 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.757468    2584 scope.go:117] "RemoveContainer" containerID="e549c7ff2471d68bc7056e81194cc2c4d953418fda599a8828cf622b08189e88"
Jul 29 21:50:42.757759 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:50:42.757741    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e549c7ff2471d68bc7056e81194cc2c4d953418fda599a8828cf622b08189e88\": container with ID starting with e549c7ff2471d68bc7056e81194cc2c4d953418fda599a8828cf622b08189e88 not found: ID does not exist" containerID="e549c7ff2471d68bc7056e81194cc2c4d953418fda599a8828cf622b08189e88"
Jul 29 21:50:42.757810 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.757766    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e549c7ff2471d68bc7056e81194cc2c4d953418fda599a8828cf622b08189e88"} err="failed to get container status \"e549c7ff2471d68bc7056e81194cc2c4d953418fda599a8828cf622b08189e88\": rpc error: code = NotFound desc = could not find container \"e549c7ff2471d68bc7056e81194cc2c4d953418fda599a8828cf622b08189e88\": container with ID starting with e549c7ff2471d68bc7056e81194cc2c4d953418fda599a8828cf622b08189e88 not found: ID does not exist"
Jul 29 21:50:42.757810 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.757785    2584 scope.go:117] "RemoveContainer" containerID="0680b9ff9a2a8f576c56cbddbda880f1288b6473fbbac6e66a8bdd334b5784fa"
Jul 29 21:50:42.758013 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:50:42.757994    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0680b9ff9a2a8f576c56cbddbda880f1288b6473fbbac6e66a8bdd334b5784fa\": container with ID starting with 0680b9ff9a2a8f576c56cbddbda880f1288b6473fbbac6e66a8bdd334b5784fa not found: ID does not exist" containerID="0680b9ff9a2a8f576c56cbddbda880f1288b6473fbbac6e66a8bdd334b5784fa"
Jul 29 21:50:42.758085 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.758023    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0680b9ff9a2a8f576c56cbddbda880f1288b6473fbbac6e66a8bdd334b5784fa"} err="failed to get container status \"0680b9ff9a2a8f576c56cbddbda880f1288b6473fbbac6e66a8bdd334b5784fa\": rpc error: code = NotFound desc = could not find container \"0680b9ff9a2a8f576c56cbddbda880f1288b6473fbbac6e66a8bdd334b5784fa\": container with ID starting with 0680b9ff9a2a8f576c56cbddbda880f1288b6473fbbac6e66a8bdd334b5784fa not found: ID does not exist"
Jul 29 21:50:42.761230 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.761208    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"]
Jul 29 21:50:42.764902 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.764883    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-25008f-openshift-default-5b5d754fb8-j8b2b"]
Jul 29 21:50:42.865766 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:42.865735    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" path="/var/lib/kubelet/pods/cd7d37bd-2365-49d5-bf89-23f296b8b1e3/volumes"
Jul 29 21:50:52.787995 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:52.787900    2584 generic.go:356] "Generic (PLEG): container finished" podID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" containerID="e4802ff0b4bcadf991d3b29b76f0e0312eadc4412f90bdf00cd5614f7245c41c" exitCode=0
Jul 29 21:50:52.787995 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:52.787972    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt" event={"ID":"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed","Type":"ContainerDied","Data":"e4802ff0b4bcadf991d3b29b76f0e0312eadc4412f90bdf00cd5614f7245c41c"}
Jul 29 21:50:52.788470 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:52.788370    2584 scope.go:117] "RemoveContainer" containerID="e4802ff0b4bcadf991d3b29b76f0e0312eadc4412f90bdf00cd5614f7245c41c"
Jul 29 21:50:52.789327 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:52.789303    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:52.789404 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:52.789369    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:52.789443 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:52.789401    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:53.797161 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:53.797102    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt" event={"ID":"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed","Type":"ContainerStarted","Data":"d81aa21c28770cdd837d1a2eb2c0e240b8f0a763da17e4c2017e19424be7bbea"}
Jul 29 21:50:53.962788 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:53.962747    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:53.963875 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:53.963836    2584 patch_prober.go:28] interesting pod/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 29 21:50:53.964005 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:53.963899    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt" podUID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 29 21:50:54.963325 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:54.963292    2584 patch_prober.go:28] interesting pod/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 29 21:50:54.963718 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:54.963343    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt" podUID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 29 21:50:55.727906 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:55.727831    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:55.962509 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:55.962472    2584 patch_prober.go:28] interesting pod/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 29 21:50:55.962691 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:55.962523    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt" podUID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 29 21:50:56.962801 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:56.962763    2584 patch_prober.go:28] interesting pod/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 29 21:50:56.963247 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:56.962819    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt" podUID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 29 21:50:57.966410 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:57.966381    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:50:57.967222 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:50:57.967204    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 21:51:06.293445 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.293413    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"]
Jul 29 21:51:06.293826 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.293808    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerName="istio-proxy"
Jul 29 21:51:06.293891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.293828    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerName="istio-proxy"
Jul 29 21:51:06.293891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.293847    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerName="istio-proxy"
Jul 29 21:51:06.293891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.293856    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerName="istio-proxy"
Jul 29 21:51:06.293891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.293880    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db92c96-14af-413b-b86b-428f3dc3d2dd" containerName="istio-proxy"
Jul 29 21:51:06.293891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.293889    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db92c96-14af-413b-b86b-428f3dc3d2dd" containerName="istio-proxy"
Jul 29 21:51:06.294093 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.293961    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db92c96-14af-413b-b86b-428f3dc3d2dd" containerName="istio-proxy"
Jul 29 21:51:06.294093 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.293978    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerName="istio-proxy"
Jul 29 21:51:06.294093 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.293986    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7d37bd-2365-49d5-bf89-23f296b8b1e3" containerName="istio-proxy"
Jul 29 21:51:06.296942 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.296927    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.300453 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.300427    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-ac47f7-openshift-default-dockercfg-wjjhl\""
Jul 29 21:51:06.308081 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.308059    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"]
Jul 29 21:51:06.407094 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.407056    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-workload-socket\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.407407 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.407188    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-credential-socket\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.407407 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.407275    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-envoy\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.407407 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.407306    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-token\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.407407 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.407393    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-workload-certs\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.407692 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.407456    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtsp4\" (UniqueName: \"kubernetes.io/projected/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-kube-api-access-vtsp4\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.407692 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.407525    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-data\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.407692 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.407553    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-podinfo\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.407692 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.407585    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istiod-ca-cert\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.508354 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.508314    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-credential-socket\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.508550 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.508367    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-envoy\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.508550 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.508385    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-token\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.508550 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.508422    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-workload-certs\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.508550 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.508445    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtsp4\" (UniqueName: \"kubernetes.io/projected/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-kube-api-access-vtsp4\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.508550 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.508467    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-data\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.508824 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.508651    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-podinfo\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.508824 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.508811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istiod-ca-cert\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.508939 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.508827    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-credential-socket\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.508939 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.508847    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-workload-socket\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.508939 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.508861    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-data\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.508939 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.508884    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-workload-certs\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.509195 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.509171    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-workload-socket\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.509463 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.509441    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istiod-ca-cert\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.510651 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.510629    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-envoy\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.510914 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.510896    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-podinfo\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.516192 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.516171    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtsp4\" (UniqueName: \"kubernetes.io/projected/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-kube-api-access-vtsp4\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.516562 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.516543    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-token\") pod \"e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") " pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.607418 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.607386    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:06.738982 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.738954    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"]
Jul 29 21:51:06.741344 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:51:06.741321    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701b1061_a6b2_45d2_bd2e_e05dd32a7f5f.slice/crio-bd7c2f7c695a456ce5f90b9914ded7ce255015264388a9953cad336869da04d0 WatchSource:0}: Error finding container bd7c2f7c695a456ce5f90b9914ded7ce255015264388a9953cad336869da04d0: Status 404 returned error can't find the container with id bd7c2f7c695a456ce5f90b9914ded7ce255015264388a9953cad336869da04d0
Jul 29 21:51:06.743926 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.743897    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:51:06.744001 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.743962    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:51:06.744001 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.743995    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:51:06.865907 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.865816    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7" event={"ID":"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f","Type":"ContainerStarted","Data":"d7a20ac010025973bd059176e625a215f9733b615e1b70b7b5b72ecd4c2b1dd2"}
Jul 29 21:51:06.865907 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.865867    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7" event={"ID":"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f","Type":"ContainerStarted","Data":"bd7c2f7c695a456ce5f90b9914ded7ce255015264388a9953cad336869da04d0"}
Jul 29 21:51:06.881233 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:06.881175    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7" podStartSLOduration=0.88112946 podStartE2EDuration="881.12946ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:51:06.880003567 +0000 UTC m=+3882.476295251" watchObservedRunningTime="2026-07-29 21:51:06.88112946 +0000 UTC m=+3882.477421144"
Jul 29 21:51:07.608338 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:07.608305    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:07.613102 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:07.613078    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:07.866665 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:07.866568    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:07.867478 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:07.867458    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:23.909352 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:23.909309    2584 patch_prober.go:28] interesting pod/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 29 21:51:23.911573 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:23.909399    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7" podUID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 21:51:24.947113 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:24.947079    2584 generic.go:356] "Generic (PLEG): container finished" podID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerID="d7a20ac010025973bd059176e625a215f9733b615e1b70b7b5b72ecd4c2b1dd2" exitCode=0
Jul 29 21:51:24.947517 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:24.947168    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7" event={"ID":"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f","Type":"ContainerDied","Data":"d7a20ac010025973bd059176e625a215f9733b615e1b70b7b5b72ecd4c2b1dd2"}
Jul 29 21:51:24.947575 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:24.947536    2584 scope.go:117] "RemoveContainer" containerID="d7a20ac010025973bd059176e625a215f9733b615e1b70b7b5b72ecd4c2b1dd2"
Jul 29 21:51:24.948503 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:24.948478    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:51:24.948564 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:24.948542    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:51:24.948615 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:24.948572    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:51:25.333860 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:25.333776    2584 scope.go:117] "RemoveContainer" containerID="a389dc4415922edfd13730c805a004dfed3edb48b95aef7decf93f45aa321822"
Jul 29 21:51:25.953044 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:25.953008    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7" event={"ID":"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f","Type":"ContainerStarted","Data":"842af4df9891350ac834d50e3d325c12889aa1aeb058ad5122f1ffafa097ba9a"}
Jul 29 21:51:26.607938 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:26.607889    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:26.608195 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:26.608037    2584 patch_prober.go:28] interesting pod/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 29 21:51:26.608195 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:26.608085    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7" podUID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 29 21:51:27.608536 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:27.608495    2584 patch_prober.go:28] interesting pod/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 29 21:51:27.609219 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:27.609183    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7" podUID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 29 21:51:28.607906 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:28.607864    2584 patch_prober.go:28] interesting pod/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 29 21:51:28.608262 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:28.608220    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7" podUID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 29 21:51:29.607966 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:29.607925    2584 patch_prober.go:28] interesting pod/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 29 21:51:29.609025 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:29.608989    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7" podUID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 29 21:51:30.611611 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:30.611582    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:30.611969 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:30.611929    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:30.612534 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:30.612512    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 21:51:36.746991 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.746949    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"]
Jul 29 21:51:36.751842 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.751822    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.753958 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.753935    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b69e52\"/\"openshift-service-ca.crt\""
Jul 29 21:51:36.754466 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.754447    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b69e52\"/\"default-dockercfg-hvgqc\""
Jul 29 21:51:36.754855 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.754839    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b69e52\"/\"kube-root-ca.crt\""
Jul 29 21:51:36.754855 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.754846    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b69e52\"/\"auth-test-model-b69e52-kserve-self-signed-certs\""
Jul 29 21:51:36.761280 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.761256    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"]
Jul 29 21:51:36.868405 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.868379    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84256\" (UniqueName: \"kubernetes.io/projected/f8826235-605e-41c7-afcc-9906b7ba23a2-kube-api-access-84256\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.868546 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.868411    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-home\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.868546 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.868461    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-tmp-dir\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.868546 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.868502    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-dshm\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.868546 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.868518    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-model-cache\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.868701 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.868571    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8826235-605e-41c7-afcc-9906b7ba23a2-tls-certs\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.969266 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.969237    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-tmp-dir\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.969423 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.969276    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-dshm\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.969423 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.969294    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-model-cache\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.969423 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.969314    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8826235-605e-41c7-afcc-9906b7ba23a2-tls-certs\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.969423 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.969331    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84256\" (UniqueName: \"kubernetes.io/projected/f8826235-605e-41c7-afcc-9906b7ba23a2-kube-api-access-84256\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.969423 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.969352    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-home\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.969707 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.969665    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-model-cache\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.969763 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.969707    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-tmp-dir\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.969763 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.969750    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-home\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.971539 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.971517    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-dshm\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.971888 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.971870    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8826235-605e-41c7-afcc-9906b7ba23a2-tls-certs\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:36.977175 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:36.977157    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84256\" (UniqueName: \"kubernetes.io/projected/f8826235-605e-41c7-afcc-9906b7ba23a2-kube-api-access-84256\") pod \"auth-test-model-b69e52-kserve-c6bd85b7-djm6t\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") " pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:37.062051 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:37.061968    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:37.190577 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:37.190552    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"]
Jul 29 21:51:37.192111 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:51:37.192085    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8826235_605e_41c7_afcc_9906b7ba23a2.slice/crio-6c13699756c5d69ce8ab7c40f7859724777fd87d0e178f10f30230eb773f1791 WatchSource:0}: Error finding container 6c13699756c5d69ce8ab7c40f7859724777fd87d0e178f10f30230eb773f1791: Status 404 returned error can't find the container with id 6c13699756c5d69ce8ab7c40f7859724777fd87d0e178f10f30230eb773f1791
Jul 29 21:51:38.006881 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:38.006848    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t" event={"ID":"f8826235-605e-41c7-afcc-9906b7ba23a2","Type":"ContainerStarted","Data":"130718f409bdc835c5963f8118e9dd555cac815a640751353cd5cfa03d7fd8f1"}
Jul 29 21:51:38.006881 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:38.006887    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t" event={"ID":"f8826235-605e-41c7-afcc-9906b7ba23a2","Type":"ContainerStarted","Data":"6c13699756c5d69ce8ab7c40f7859724777fd87d0e178f10f30230eb773f1791"}
Jul 29 21:51:38.027867 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:38.027825    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t" podStartSLOduration=2.027811349 podStartE2EDuration="2.027811349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:51:38.025716321 +0000 UTC m=+3913.622008005" watchObservedRunningTime="2026-07-29 21:51:38.027811349 +0000 UTC m=+3913.624103032"
Jul 29 21:51:47.062757 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:47.062721    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:47.063216 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:47.062773    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:47.075137 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:47.075116    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:48.060796 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:48.060770    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:51:54.347531 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.347498    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"]
Jul 29 21:51:54.351102 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.351086    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.353470 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.353445    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac47f7\"/\"default-dockercfg-bcjjf\""
Jul 29 21:51:54.353591 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.353483    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac47f7\"/\"openshift-service-ca.crt\""
Jul 29 21:51:54.354096 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.354072    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac47f7\"/\"auth-test-model-ac47f7-kserve-self-signed-certs\""
Jul 29 21:51:54.354192 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.354172    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac47f7\"/\"kube-root-ca.crt\""
Jul 29 21:51:54.359135 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.359115    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"]
Jul 29 21:51:54.413624 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.413597    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-model-cache\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.413759 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.413627    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-dshm\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.413759 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.413670    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-home\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.413759 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.413742    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-tmp-dir\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.413877 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.413764    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdq6d\" (UniqueName: \"kubernetes.io/projected/f10298c3-afa8-4e7e-92aa-6529224f5efb-kube-api-access-qdq6d\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.413877 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.413783    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10298c3-afa8-4e7e-92aa-6529224f5efb-tls-certs\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.514450 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.514423    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-tmp-dir\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.514599 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.514456    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdq6d\" (UniqueName: \"kubernetes.io/projected/f10298c3-afa8-4e7e-92aa-6529224f5efb-kube-api-access-qdq6d\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.514599 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.514475    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10298c3-afa8-4e7e-92aa-6529224f5efb-tls-certs\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.514599 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.514497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-model-cache\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.514740 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.514623    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-dshm\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.514740 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.514673    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-home\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.514891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.514858    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-tmp-dir\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.515005 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.514944    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-model-cache\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.515057 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.515020    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-home\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.516732 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.516712    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-dshm\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.517117 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.517096    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10298c3-afa8-4e7e-92aa-6529224f5efb-tls-certs\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.521923 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.521902    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdq6d\" (UniqueName: \"kubernetes.io/projected/f10298c3-afa8-4e7e-92aa-6529224f5efb-kube-api-access-qdq6d\") pod \"auth-test-model-ac47f7-kserve-75d4596c9f-fbm77\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") " pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.661882 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.661805    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:51:54.789481 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:54.789454    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"]
Jul 29 21:51:54.792243 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:51:54.792219    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf10298c3_afa8_4e7e_92aa_6529224f5efb.slice/crio-df7662dbaa11d9aa58cf94b5de4c274c91dd827c461f0a0ba034fba4b6eb74ff WatchSource:0}: Error finding container df7662dbaa11d9aa58cf94b5de4c274c91dd827c461f0a0ba034fba4b6eb74ff: Status 404 returned error can't find the container with id df7662dbaa11d9aa58cf94b5de4c274c91dd827c461f0a0ba034fba4b6eb74ff
Jul 29 21:51:55.082327 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:55.082231    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77" event={"ID":"f10298c3-afa8-4e7e-92aa-6529224f5efb","Type":"ContainerStarted","Data":"8322f1906718721eb2ffca47872018366a4d518ead92a1e6d1a157b5182fae31"}
Jul 29 21:51:55.082327 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:55.082280    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77" event={"ID":"f10298c3-afa8-4e7e-92aa-6529224f5efb","Type":"ContainerStarted","Data":"df7662dbaa11d9aa58cf94b5de4c274c91dd827c461f0a0ba034fba4b6eb74ff"}
Jul 29 21:51:55.118081 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:51:55.118032    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77" podStartSLOduration=1.118017816 podStartE2EDuration="1.118017816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:51:55.11588277 +0000 UTC m=+3930.712174454" watchObservedRunningTime="2026-07-29 21:51:55.118017816 +0000 UTC m=+3930.714309500"
Jul 29 21:52:04.662651 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:52:04.662610    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:52:04.662651 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:52:04.662661    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:52:04.676905 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:52:04.676876    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:52:05.139648 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:52:05.139617    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:56:25.384372 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:56:25.384341    2584 scope.go:117] "RemoveContainer" containerID="cfa8697ff7b5dbca07bc06b49ed1fb9338c755e6dcb954477bd7b340f42c83bf"
Jul 29 21:56:56.673387 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:56:56.673309    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"]
Jul 29 21:56:57.519467 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:56:57.519424    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"]
Jul 29 21:57:11.691999 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.691942    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t" podUID="f8826235-605e-41c7-afcc-9906b7ba23a2" containerName="main" containerID="cri-o://130718f409bdc835c5963f8118e9dd555cac815a640751353cd5cfa03d7fd8f1" gracePeriod=45
Jul 29 21:57:11.945795 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.945743    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:57:11.990503 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.990474    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84256\" (UniqueName: \"kubernetes.io/projected/f8826235-605e-41c7-afcc-9906b7ba23a2-kube-api-access-84256\") pod \"f8826235-605e-41c7-afcc-9906b7ba23a2\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") "
Jul 29 21:57:11.990629 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.990523    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-dshm\") pod \"f8826235-605e-41c7-afcc-9906b7ba23a2\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") "
Jul 29 21:57:11.990629 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.990552    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-home\") pod \"f8826235-605e-41c7-afcc-9906b7ba23a2\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") "
Jul 29 21:57:11.990629 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.990586    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8826235-605e-41c7-afcc-9906b7ba23a2-tls-certs\") pod \"f8826235-605e-41c7-afcc-9906b7ba23a2\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") "
Jul 29 21:57:11.990629 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.990627    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-model-cache\") pod \"f8826235-605e-41c7-afcc-9906b7ba23a2\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") "
Jul 29 21:57:11.990853 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.990708    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-tmp-dir\") pod \"f8826235-605e-41c7-afcc-9906b7ba23a2\" (UID: \"f8826235-605e-41c7-afcc-9906b7ba23a2\") "
Jul 29 21:57:11.990853 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.990838    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-home" (OuterVolumeSpecName: "home") pod "f8826235-605e-41c7-afcc-9906b7ba23a2" (UID: "f8826235-605e-41c7-afcc-9906b7ba23a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:57:11.990998 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.990967    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:57:11.991057 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.990975    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8826235-605e-41c7-afcc-9906b7ba23a2" (UID: "f8826235-605e-41c7-afcc-9906b7ba23a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:57:11.991116 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.991054    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8826235-605e-41c7-afcc-9906b7ba23a2" (UID: "f8826235-605e-41c7-afcc-9906b7ba23a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:57:11.992753 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.992718    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8826235-605e-41c7-afcc-9906b7ba23a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8826235-605e-41c7-afcc-9906b7ba23a2" (UID: "f8826235-605e-41c7-afcc-9906b7ba23a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:57:11.992753 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.992728    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8826235-605e-41c7-afcc-9906b7ba23a2-kube-api-access-84256" (OuterVolumeSpecName: "kube-api-access-84256") pod "f8826235-605e-41c7-afcc-9906b7ba23a2" (UID: "f8826235-605e-41c7-afcc-9906b7ba23a2"). InnerVolumeSpecName "kube-api-access-84256". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:57:11.992888 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:11.992761    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-dshm" (OuterVolumeSpecName: "dshm") pod "f8826235-605e-41c7-afcc-9906b7ba23a2" (UID: "f8826235-605e-41c7-afcc-9906b7ba23a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:57:12.091519 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.091497    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:57:12.091641 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.091539    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8826235-605e-41c7-afcc-9906b7ba23a2-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:57:12.091641 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.091549    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:57:12.091641 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.091557    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8826235-605e-41c7-afcc-9906b7ba23a2-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:57:12.091641 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.091565    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84256\" (UniqueName: \"kubernetes.io/projected/f8826235-605e-41c7-afcc-9906b7ba23a2-kube-api-access-84256\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:57:12.537888 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.537830    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77" podUID="f10298c3-afa8-4e7e-92aa-6529224f5efb" containerName="main" containerID="cri-o://8322f1906718721eb2ffca47872018366a4d518ead92a1e6d1a157b5182fae31" gracePeriod=45
Jul 29 21:57:12.543921 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.543895    2584 generic.go:356] "Generic (PLEG): container finished" podID="f8826235-605e-41c7-afcc-9906b7ba23a2" containerID="130718f409bdc835c5963f8118e9dd555cac815a640751353cd5cfa03d7fd8f1" exitCode=0
Jul 29 21:57:12.544029 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.543933    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"
Jul 29 21:57:12.544029 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.543949    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t" event={"ID":"f8826235-605e-41c7-afcc-9906b7ba23a2","Type":"ContainerDied","Data":"130718f409bdc835c5963f8118e9dd555cac815a640751353cd5cfa03d7fd8f1"}
Jul 29 21:57:12.544029 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.543988    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t" event={"ID":"f8826235-605e-41c7-afcc-9906b7ba23a2","Type":"ContainerDied","Data":"6c13699756c5d69ce8ab7c40f7859724777fd87d0e178f10f30230eb773f1791"}
Jul 29 21:57:12.544029 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.544006    2584 scope.go:117] "RemoveContainer" containerID="130718f409bdc835c5963f8118e9dd555cac815a640751353cd5cfa03d7fd8f1"
Jul 29 21:57:12.595158 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.595120    2584 scope.go:117] "RemoveContainer" containerID="130718f409bdc835c5963f8118e9dd555cac815a640751353cd5cfa03d7fd8f1"
Jul 29 21:57:12.595466 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:57:12.595436    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"130718f409bdc835c5963f8118e9dd555cac815a640751353cd5cfa03d7fd8f1\": container with ID starting with 130718f409bdc835c5963f8118e9dd555cac815a640751353cd5cfa03d7fd8f1 not found: ID does not exist" containerID="130718f409bdc835c5963f8118e9dd555cac815a640751353cd5cfa03d7fd8f1"
Jul 29 21:57:12.595545 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.595477    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"130718f409bdc835c5963f8118e9dd555cac815a640751353cd5cfa03d7fd8f1"} err="failed to get container status \"130718f409bdc835c5963f8118e9dd555cac815a640751353cd5cfa03d7fd8f1\": rpc error: code = NotFound desc = could not find container \"130718f409bdc835c5963f8118e9dd555cac815a640751353cd5cfa03d7fd8f1\": container with ID starting with 130718f409bdc835c5963f8118e9dd555cac815a640751353cd5cfa03d7fd8f1 not found: ID does not exist"
Jul 29 21:57:12.605406 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.605385    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"]
Jul 29 21:57:12.608877 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.608853    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/auth-test-model-b69e52-kserve-c6bd85b7-djm6t"]
Jul 29 21:57:12.790790 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.790737    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:57:12.796658 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.796640    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-home\") pod \"f10298c3-afa8-4e7e-92aa-6529224f5efb\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") "
Jul 29 21:57:12.796736 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.796672    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10298c3-afa8-4e7e-92aa-6529224f5efb-tls-certs\") pod \"f10298c3-afa8-4e7e-92aa-6529224f5efb\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") "
Jul 29 21:57:12.796736 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.796699    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-model-cache\") pod \"f10298c3-afa8-4e7e-92aa-6529224f5efb\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") "
Jul 29 21:57:12.796736 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.796729    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdq6d\" (UniqueName: \"kubernetes.io/projected/f10298c3-afa8-4e7e-92aa-6529224f5efb-kube-api-access-qdq6d\") pod \"f10298c3-afa8-4e7e-92aa-6529224f5efb\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") "
Jul 29 21:57:12.796885 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.796758    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-tmp-dir\") pod \"f10298c3-afa8-4e7e-92aa-6529224f5efb\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") "
Jul 29 21:57:12.796885 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.796781    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-dshm\") pod \"f10298c3-afa8-4e7e-92aa-6529224f5efb\" (UID: \"f10298c3-afa8-4e7e-92aa-6529224f5efb\") "
Jul 29 21:57:12.796993 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.796894    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-home" (OuterVolumeSpecName: "home") pod "f10298c3-afa8-4e7e-92aa-6529224f5efb" (UID: "f10298c3-afa8-4e7e-92aa-6529224f5efb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:57:12.796993 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.796908    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-model-cache" (OuterVolumeSpecName: "model-cache") pod "f10298c3-afa8-4e7e-92aa-6529224f5efb" (UID: "f10298c3-afa8-4e7e-92aa-6529224f5efb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:57:12.797114 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.797023    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f10298c3-afa8-4e7e-92aa-6529224f5efb" (UID: "f10298c3-afa8-4e7e-92aa-6529224f5efb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:57:12.797114 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.797061    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:57:12.797114 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.797074    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:57:12.798783 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.798764    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-dshm" (OuterVolumeSpecName: "dshm") pod "f10298c3-afa8-4e7e-92aa-6529224f5efb" (UID: "f10298c3-afa8-4e7e-92aa-6529224f5efb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:57:12.798891 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.798873    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10298c3-afa8-4e7e-92aa-6529224f5efb-kube-api-access-qdq6d" (OuterVolumeSpecName: "kube-api-access-qdq6d") pod "f10298c3-afa8-4e7e-92aa-6529224f5efb" (UID: "f10298c3-afa8-4e7e-92aa-6529224f5efb"). InnerVolumeSpecName "kube-api-access-qdq6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:57:12.798939 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.798873    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10298c3-afa8-4e7e-92aa-6529224f5efb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f10298c3-afa8-4e7e-92aa-6529224f5efb" (UID: "f10298c3-afa8-4e7e-92aa-6529224f5efb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:57:12.865584 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.865559    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8826235-605e-41c7-afcc-9906b7ba23a2" path="/var/lib/kubelet/pods/f8826235-605e-41c7-afcc-9906b7ba23a2/volumes"
Jul 29 21:57:12.898194 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.898171    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdq6d\" (UniqueName: \"kubernetes.io/projected/f10298c3-afa8-4e7e-92aa-6529224f5efb-kube-api-access-qdq6d\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:57:12.898194 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.898193    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:57:12.898334 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.898202    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10298c3-afa8-4e7e-92aa-6529224f5efb-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:57:12.898334 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:12.898212    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10298c3-afa8-4e7e-92aa-6529224f5efb-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:57:13.550170 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:13.550123    2584 generic.go:356] "Generic (PLEG): container finished" podID="f10298c3-afa8-4e7e-92aa-6529224f5efb" containerID="8322f1906718721eb2ffca47872018366a4d518ead92a1e6d1a157b5182fae31" exitCode=0
Jul 29 21:57:13.550362 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:13.550192    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"
Jul 29 21:57:13.550362 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:13.550205    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77" event={"ID":"f10298c3-afa8-4e7e-92aa-6529224f5efb","Type":"ContainerDied","Data":"8322f1906718721eb2ffca47872018366a4d518ead92a1e6d1a157b5182fae31"}
Jul 29 21:57:13.550362 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:13.550248    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77" event={"ID":"f10298c3-afa8-4e7e-92aa-6529224f5efb","Type":"ContainerDied","Data":"df7662dbaa11d9aa58cf94b5de4c274c91dd827c461f0a0ba034fba4b6eb74ff"}
Jul 29 21:57:13.550362 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:13.550266    2584 scope.go:117] "RemoveContainer" containerID="8322f1906718721eb2ffca47872018366a4d518ead92a1e6d1a157b5182fae31"
Jul 29 21:57:13.558500 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:13.558484    2584 scope.go:117] "RemoveContainer" containerID="8322f1906718721eb2ffca47872018366a4d518ead92a1e6d1a157b5182fae31"
Jul 29 21:57:13.558741 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:57:13.558722    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8322f1906718721eb2ffca47872018366a4d518ead92a1e6d1a157b5182fae31\": container with ID starting with 8322f1906718721eb2ffca47872018366a4d518ead92a1e6d1a157b5182fae31 not found: ID does not exist" containerID="8322f1906718721eb2ffca47872018366a4d518ead92a1e6d1a157b5182fae31"
Jul 29 21:57:13.558785 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:13.558757    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8322f1906718721eb2ffca47872018366a4d518ead92a1e6d1a157b5182fae31"} err="failed to get container status \"8322f1906718721eb2ffca47872018366a4d518ead92a1e6d1a157b5182fae31\": rpc error: code = NotFound desc = could not find container \"8322f1906718721eb2ffca47872018366a4d518ead92a1e6d1a157b5182fae31\": container with ID starting with 8322f1906718721eb2ffca47872018366a4d518ead92a1e6d1a157b5182fae31 not found: ID does not exist"
Jul 29 21:57:13.566615 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:13.566594    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"]
Jul 29 21:57:13.571275 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:13.571255    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ac47f7/auth-test-model-ac47f7-kserve-75d4596c9f-fbm77"]
Jul 29 21:57:14.349170 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.349128    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"]
Jul 29 21:57:14.349529 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.349466    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8826235-605e-41c7-afcc-9906b7ba23a2" containerName="main"
Jul 29 21:57:14.349529 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.349478    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8826235-605e-41c7-afcc-9906b7ba23a2" containerName="main"
Jul 29 21:57:14.349529 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.349502    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f10298c3-afa8-4e7e-92aa-6529224f5efb" containerName="main"
Jul 29 21:57:14.349529 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.349508    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f10298c3-afa8-4e7e-92aa-6529224f5efb" containerName="main"
Jul 29 21:57:14.349667 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.349558    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8826235-605e-41c7-afcc-9906b7ba23a2" containerName="main"
Jul 29 21:57:14.349667 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.349567    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f10298c3-afa8-4e7e-92aa-6529224f5efb" containerName="main"
Jul 29 21:57:14.352339 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.352323    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.354786 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.354761    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b69e52\"/\"openshift-service-ca.crt\""
Jul 29 21:57:14.354786 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.354778    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b69e52\"/\"default-dockercfg-hvgqc\""
Jul 29 21:57:14.355021 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.354841    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b69e52\"/\"sub-test-model-b69e52-kserve-self-signed-certs\""
Jul 29 21:57:14.355021 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.354863    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b69e52\"/\"kube-root-ca.crt\""
Jul 29 21:57:14.362039 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.362016    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"]
Jul 29 21:57:14.410013 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.409986    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhw28\" (UniqueName: \"kubernetes.io/projected/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-kube-api-access-qhw28\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.410013 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.410016    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-home\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.410273 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.410040    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-tls-certs\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.410273 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.410169    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-tmp-dir\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.410273 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.410211    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-model-cache\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.410273 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.410244    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-dshm\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.510745 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.510710    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-model-cache\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.510745 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.510748    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-dshm\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.510957 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.510786    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhw28\" (UniqueName: \"kubernetes.io/projected/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-kube-api-access-qhw28\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.510957 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.510807    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-home\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.510957 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.510831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-tls-certs\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.510957 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.510865    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-tmp-dir\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.511209 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.511188    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-model-cache\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.511263 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.511227    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-home\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.511308 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.511290    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-tmp-dir\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.513110 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.513088    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-dshm\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.513376 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.513356    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-tls-certs\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.518479 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.518458    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhw28\" (UniqueName: \"kubernetes.io/projected/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-kube-api-access-qhw28\") pod \"sub-test-model-b69e52-kserve-db59fd75c-grbhv\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") " pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.661303 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.661205    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:14.789350 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.789318    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"]
Jul 29 21:57:14.792806 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:57:14.792780    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7611b30_6661_4f9d_9a0f_1da22d8b1b7b.slice/crio-7f70b40039de76f2b31768e6a3935aaed6f12282cf954908e1ea6590c4a06ed5 WatchSource:0}: Error finding container 7f70b40039de76f2b31768e6a3935aaed6f12282cf954908e1ea6590c4a06ed5: Status 404 returned error can't find the container with id 7f70b40039de76f2b31768e6a3935aaed6f12282cf954908e1ea6590c4a06ed5
Jul 29 21:57:14.867043 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:14.867014    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f10298c3-afa8-4e7e-92aa-6529224f5efb" path="/var/lib/kubelet/pods/f10298c3-afa8-4e7e-92aa-6529224f5efb/volumes"
Jul 29 21:57:15.562846 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:15.562811    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv" event={"ID":"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b","Type":"ContainerStarted","Data":"4c50f9f56f1478c49169a9804bb4c9c726886725e57af514ea8d9d147e79e45d"}
Jul 29 21:57:15.563246 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:15.562856    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv" event={"ID":"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b","Type":"ContainerStarted","Data":"7f70b40039de76f2b31768e6a3935aaed6f12282cf954908e1ea6590c4a06ed5"}
Jul 29 21:57:15.584396 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:15.584354    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv" podStartSLOduration=1.584340144 podStartE2EDuration="1.584340144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:57:15.583398748 +0000 UTC m=+4251.179690433" watchObservedRunningTime="2026-07-29 21:57:15.584340144 +0000 UTC m=+4251.180631822"
Jul 29 21:57:24.662120 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:24.662077    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:24.662568 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:24.662132    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:24.674919 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:24.674891    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:57:25.617360 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:57:25.617325    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:59:43.212837 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:43.212801    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"]
Jul 29 21:59:55.952600 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:55.952569    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"]
Jul 29 21:59:55.958046 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:55.958016    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:55.960395 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:55.960365    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b69e52\"/\"rate-test-model-b69e52-kserve-self-signed-certs\""
Jul 29 21:59:55.965487 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:55.965459    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"]
Jul 29 21:59:56.129567 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.129531    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-dshm\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.129567 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.129571    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-home\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.129774 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.129594    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-tmp-dir\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.129774 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.129701    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-tls-certs\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.129774 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.129740    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-model-cache\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.129872 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.129798    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72rk8\" (UniqueName: \"kubernetes.io/projected/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-kube-api-access-72rk8\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.230694 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.230618    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-dshm\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.230694 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.230651    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-home\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.230694 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.230670    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-tmp-dir\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.230973 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.230700    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-tls-certs\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.230973 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.230837    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-model-cache\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.230973 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.230915    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72rk8\" (UniqueName: \"kubernetes.io/projected/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-kube-api-access-72rk8\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.231126 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.231040    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-home\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.231217 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.231171    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-tmp-dir\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.231320 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.231298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-model-cache\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.233045 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.233025    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-dshm\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.233249 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.233232    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-tls-certs\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.239210 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.239186    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72rk8\" (UniqueName: \"kubernetes.io/projected/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-kube-api-access-72rk8\") pod \"rate-test-model-b69e52-kserve-d4d6b5687-49jq4\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") " pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.269035 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.269013    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 21:59:56.398010 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:56.397983    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"]
Jul 29 21:59:56.400120 ip-10-0-129-194 kubenswrapper[2584]: W0729 21:59:56.400093    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cc377ca_cb0f_4bd1_a08b_7f700879a4f7.slice/crio-2f61b152f8a091886dc3edf69db2c8ef3a9aeaf4afb2999050ca216f3f3bdfa6 WatchSource:0}: Error finding container 2f61b152f8a091886dc3edf69db2c8ef3a9aeaf4afb2999050ca216f3f3bdfa6: Status 404 returned error can't find the container with id 2f61b152f8a091886dc3edf69db2c8ef3a9aeaf4afb2999050ca216f3f3bdfa6
Jul 29 21:59:57.279079 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:57.279039    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4" event={"ID":"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7","Type":"ContainerStarted","Data":"0113b3012486920365e0f4fc13e200a6ba3b7ab4f87bf866c7aa0ccf3ffc7d52"}
Jul 29 21:59:57.279479 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:57.279088    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4" event={"ID":"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7","Type":"ContainerStarted","Data":"2f61b152f8a091886dc3edf69db2c8ef3a9aeaf4afb2999050ca216f3f3bdfa6"}
Jul 29 21:59:57.297949 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:57.297906    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4" podStartSLOduration=2.297890885 podStartE2EDuration="2.297890885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:59:57.29689968 +0000 UTC m=+4412.893191364" watchObservedRunningTime="2026-07-29 21:59:57.297890885 +0000 UTC m=+4412.894182568"
Jul 29 21:59:58.231554 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.231513    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv" podUID="d7611b30-6661-4f9d-9a0f-1da22d8b1b7b" containerName="main" containerID="cri-o://4c50f9f56f1478c49169a9804bb4c9c726886725e57af514ea8d9d147e79e45d" gracePeriod=45
Jul 29 21:59:58.481620 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.481561    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:59:58.655626 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.655592    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-tls-certs\") pod \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") "
Jul 29 21:59:58.655808 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.655636    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-dshm\") pod \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") "
Jul 29 21:59:58.655808 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.655663    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-model-cache\") pod \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") "
Jul 29 21:59:58.655808 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.655693    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-home\") pod \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") "
Jul 29 21:59:58.655808 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.655714    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-tmp-dir\") pod \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") "
Jul 29 21:59:58.655808 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.655737    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhw28\" (UniqueName: \"kubernetes.io/projected/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-kube-api-access-qhw28\") pod \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\" (UID: \"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b\") "
Jul 29 21:59:58.656078 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.655969    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d7611b30-6661-4f9d-9a0f-1da22d8b1b7b" (UID: "d7611b30-6661-4f9d-9a0f-1da22d8b1b7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:59:58.656078 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.656009    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-home" (OuterVolumeSpecName: "home") pod "d7611b30-6661-4f9d-9a0f-1da22d8b1b7b" (UID: "d7611b30-6661-4f9d-9a0f-1da22d8b1b7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:59:58.656078 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.656042    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d7611b30-6661-4f9d-9a0f-1da22d8b1b7b" (UID: "d7611b30-6661-4f9d-9a0f-1da22d8b1b7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:59:58.657833 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.657804    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7611b30-6661-4f9d-9a0f-1da22d8b1b7b" (UID: "d7611b30-6661-4f9d-9a0f-1da22d8b1b7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:59:58.657957 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.657831    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-dshm" (OuterVolumeSpecName: "dshm") pod "d7611b30-6661-4f9d-9a0f-1da22d8b1b7b" (UID: "d7611b30-6661-4f9d-9a0f-1da22d8b1b7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:59:58.657957 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.657878    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-kube-api-access-qhw28" (OuterVolumeSpecName: "kube-api-access-qhw28") pod "d7611b30-6661-4f9d-9a0f-1da22d8b1b7b" (UID: "d7611b30-6661-4f9d-9a0f-1da22d8b1b7b"). InnerVolumeSpecName "kube-api-access-qhw28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:59:58.756647 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.756577    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:59:58.756647 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.756602    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:59:58.756647 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.756611    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:59:58.756647 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.756619    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:59:58.756647 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.756628    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:59:58.756647 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:58.756637    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhw28\" (UniqueName: \"kubernetes.io/projected/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b-kube-api-access-qhw28\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 21:59:59.293096 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:59.293058    2584 generic.go:356] "Generic (PLEG): container finished" podID="d7611b30-6661-4f9d-9a0f-1da22d8b1b7b" containerID="4c50f9f56f1478c49169a9804bb4c9c726886725e57af514ea8d9d147e79e45d" exitCode=0
Jul 29 21:59:59.293096 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:59.293096    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"
Jul 29 21:59:59.293322 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:59.293156    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv" event={"ID":"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b","Type":"ContainerDied","Data":"4c50f9f56f1478c49169a9804bb4c9c726886725e57af514ea8d9d147e79e45d"}
Jul 29 21:59:59.293322 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:59.293201    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv" event={"ID":"d7611b30-6661-4f9d-9a0f-1da22d8b1b7b","Type":"ContainerDied","Data":"7f70b40039de76f2b31768e6a3935aaed6f12282cf954908e1ea6590c4a06ed5"}
Jul 29 21:59:59.293322 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:59.293223    2584 scope.go:117] "RemoveContainer" containerID="4c50f9f56f1478c49169a9804bb4c9c726886725e57af514ea8d9d147e79e45d"
Jul 29 21:59:59.301365 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:59.301348    2584 scope.go:117] "RemoveContainer" containerID="4c50f9f56f1478c49169a9804bb4c9c726886725e57af514ea8d9d147e79e45d"
Jul 29 21:59:59.301602 ip-10-0-129-194 kubenswrapper[2584]: E0729 21:59:59.301585    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c50f9f56f1478c49169a9804bb4c9c726886725e57af514ea8d9d147e79e45d\": container with ID starting with 4c50f9f56f1478c49169a9804bb4c9c726886725e57af514ea8d9d147e79e45d not found: ID does not exist" containerID="4c50f9f56f1478c49169a9804bb4c9c726886725e57af514ea8d9d147e79e45d"
Jul 29 21:59:59.301670 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:59.301607    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c50f9f56f1478c49169a9804bb4c9c726886725e57af514ea8d9d147e79e45d"} err="failed to get container status \"4c50f9f56f1478c49169a9804bb4c9c726886725e57af514ea8d9d147e79e45d\": rpc error: code = NotFound desc = could not find container \"4c50f9f56f1478c49169a9804bb4c9c726886725e57af514ea8d9d147e79e45d\": container with ID starting with 4c50f9f56f1478c49169a9804bb4c9c726886725e57af514ea8d9d147e79e45d not found: ID does not exist"
Jul 29 21:59:59.308826 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:59.308804    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"]
Jul 29 21:59:59.312903 ip-10-0-129-194 kubenswrapper[2584]: I0729 21:59:59.312882    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/sub-test-model-b69e52-kserve-db59fd75c-grbhv"]
Jul 29 22:00:00.865893 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:00.865849    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7611b30-6661-4f9d-9a0f-1da22d8b1b7b" path="/var/lib/kubelet/pods/d7611b30-6661-4f9d-9a0f-1da22d8b1b7b/volumes"
Jul 29 22:00:04.201362 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:04.201314    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm"]
Jul 29 22:00:04.206059 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:04.206036    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755995-rrttm"]
Jul 29 22:00:04.865193 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:04.865160    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ac90060-c993-441c-b079-b124691cbc79" path="/var/lib/kubelet/pods/8ac90060-c993-441c-b079-b124691cbc79/volumes"
Jul 29 22:00:06.269529 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:06.269499    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 22:00:06.270027 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:06.269554    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 22:00:06.281628 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:06.281593    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 22:00:06.337939 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:06.337912    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 22:00:25.429683 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.429637    2584 scope.go:117] "RemoveContainer" containerID="eb6bd3b9713a4f31eb1e85b5bc126f199afe6e3e93f08a0c5f90e8b7d28fb94e"
Jul 29 22:00:25.848981 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.848950    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"]
Jul 29 22:00:25.849335 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.849321    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7611b30-6661-4f9d-9a0f-1da22d8b1b7b" containerName="main"
Jul 29 22:00:25.849408 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.849337    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7611b30-6661-4f9d-9a0f-1da22d8b1b7b" containerName="main"
Jul 29 22:00:25.849408 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.849395    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7611b30-6661-4f9d-9a0f-1da22d8b1b7b" containerName="main"
Jul 29 22:00:25.853559 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.853536    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:25.855621 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.855597    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac47f7\"/\"openshift-service-ca.crt\""
Jul 29 22:00:25.855728 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.855600    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac47f7\"/\"kube-root-ca.crt\""
Jul 29 22:00:25.856262 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.856246    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac47f7\"/\"default-dockercfg-bcjjf\""
Jul 29 22:00:25.856352 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.856279    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac47f7\"/\"rate-test-model-ac47f7-kserve-self-signed-certs\""
Jul 29 22:00:25.863117 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.863096    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"]
Jul 29 22:00:25.983347 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.983310    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-dshm\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:25.983347 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.983347    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55141f95-1bdc-4f48-acf2-8801738825bb-tls-certs\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:25.983564 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.983383    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn8pk\" (UniqueName: \"kubernetes.io/projected/55141f95-1bdc-4f48-acf2-8801738825bb-kube-api-access-fn8pk\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:25.983564 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.983480    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-home\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:25.983564 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.983514    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-tmp-dir\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:25.983564 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:25.983536    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-model-cache\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.084766 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.084726    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn8pk\" (UniqueName: \"kubernetes.io/projected/55141f95-1bdc-4f48-acf2-8801738825bb-kube-api-access-fn8pk\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.084960 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.084809    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-home\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.084960 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.084844    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-tmp-dir\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.084960 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.084873    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-model-cache\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.084960 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.084927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-dshm\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.084960 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.084948    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55141f95-1bdc-4f48-acf2-8801738825bb-tls-certs\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.085268 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.085239    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-home\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.085332 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.085309    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-tmp-dir\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.085380 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.085330    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-model-cache\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.087327 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.087290    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-dshm\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.087459 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.087400    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55141f95-1bdc-4f48-acf2-8801738825bb-tls-certs\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.092201 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.092181    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn8pk\" (UniqueName: \"kubernetes.io/projected/55141f95-1bdc-4f48-acf2-8801738825bb-kube-api-access-fn8pk\") pod \"rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") " pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.163048 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.162967    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:26.295358 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.295319    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"]
Jul 29 22:00:26.298206 ip-10-0-129-194 kubenswrapper[2584]: W0729 22:00:26.298177    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55141f95_1bdc_4f48_acf2_8801738825bb.slice/crio-1612f1b255e74902900f6f2cb5a89cb20851c95744bd8de6453b9e6bebc27218 WatchSource:0}: Error finding container 1612f1b255e74902900f6f2cb5a89cb20851c95744bd8de6453b9e6bebc27218: Status 404 returned error can't find the container with id 1612f1b255e74902900f6f2cb5a89cb20851c95744bd8de6453b9e6bebc27218
Jul 29 22:00:26.416280 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.416184    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z" event={"ID":"55141f95-1bdc-4f48-acf2-8801738825bb","Type":"ContainerStarted","Data":"749f05c5138aad75243b9ae481299194d0faee775f1f17b21f5653d702afae5a"}
Jul 29 22:00:26.416280 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.416228    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z" event={"ID":"55141f95-1bdc-4f48-acf2-8801738825bb","Type":"ContainerStarted","Data":"1612f1b255e74902900f6f2cb5a89cb20851c95744bd8de6453b9e6bebc27218"}
Jul 29 22:00:26.432723 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:26.432671    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z" podStartSLOduration=1.432656357 podStartE2EDuration="1.432656357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:00:26.430180675 +0000 UTC m=+4442.026472377" watchObservedRunningTime="2026-07-29 22:00:26.432656357 +0000 UTC m=+4442.028948041"
Jul 29 22:00:36.163858 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:36.163815    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:36.164420 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:36.163872    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:36.176627 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:36.176604    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:00:36.470207 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:00:36.470069    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:02:47.002094 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:02:47.002005    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"]
Jul 29 22:02:47.830121 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:02:47.830081    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"]
Jul 29 22:03:02.021902 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.021858    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4" podUID="5cc377ca-cb0f-4bd1-a08b-7f700879a4f7" containerName="main" containerID="cri-o://0113b3012486920365e0f4fc13e200a6ba3b7ab4f87bf866c7aa0ccf3ffc7d52" gracePeriod=45
Jul 29 22:03:02.277748 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.277682    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 22:03:02.414139 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.414101    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-dshm\") pod \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") "
Jul 29 22:03:02.414372 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.414184    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-home\") pod \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") "
Jul 29 22:03:02.414372 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.414202    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-tls-certs\") pod \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") "
Jul 29 22:03:02.414372 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.414224    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-tmp-dir\") pod \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") "
Jul 29 22:03:02.414372 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.414275    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-model-cache\") pod \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") "
Jul 29 22:03:02.414372 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.414323    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72rk8\" (UniqueName: \"kubernetes.io/projected/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-kube-api-access-72rk8\") pod \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\" (UID: \"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7\") "
Jul 29 22:03:02.414629 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.414537    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-home" (OuterVolumeSpecName: "home") pod "5cc377ca-cb0f-4bd1-a08b-7f700879a4f7" (UID: "5cc377ca-cb0f-4bd1-a08b-7f700879a4f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:02.414629 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.414571    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "5cc377ca-cb0f-4bd1-a08b-7f700879a4f7" (UID: "5cc377ca-cb0f-4bd1-a08b-7f700879a4f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:02.414629 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.414544    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5cc377ca-cb0f-4bd1-a08b-7f700879a4f7" (UID: "5cc377ca-cb0f-4bd1-a08b-7f700879a4f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:02.416376 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.416344    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-dshm" (OuterVolumeSpecName: "dshm") pod "5cc377ca-cb0f-4bd1-a08b-7f700879a4f7" (UID: "5cc377ca-cb0f-4bd1-a08b-7f700879a4f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:02.416484 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.416392    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-kube-api-access-72rk8" (OuterVolumeSpecName: "kube-api-access-72rk8") pod "5cc377ca-cb0f-4bd1-a08b-7f700879a4f7" (UID: "5cc377ca-cb0f-4bd1-a08b-7f700879a4f7"). InnerVolumeSpecName "kube-api-access-72rk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:02.416484 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.416419    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5cc377ca-cb0f-4bd1-a08b-7f700879a4f7" (UID: "5cc377ca-cb0f-4bd1-a08b-7f700879a4f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:03:02.515193 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.515128    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:02.515193 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.515185    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:02.515193 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.515197    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:02.515447 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.515212    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:02.515447 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.515224    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:02.515447 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.515235    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72rk8\" (UniqueName: \"kubernetes.io/projected/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7-kube-api-access-72rk8\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:02.849336 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:02.849298    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z" podUID="55141f95-1bdc-4f48-acf2-8801738825bb" containerName="main" containerID="cri-o://749f05c5138aad75243b9ae481299194d0faee775f1f17b21f5653d702afae5a" gracePeriod=45
Jul 29 22:03:03.100818 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.100753    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:03:03.110344 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.110316    2584 generic.go:356] "Generic (PLEG): container finished" podID="55141f95-1bdc-4f48-acf2-8801738825bb" containerID="749f05c5138aad75243b9ae481299194d0faee775f1f17b21f5653d702afae5a" exitCode=0
Jul 29 22:03:03.110485 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.110357    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"
Jul 29 22:03:03.110485 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.110399    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z" event={"ID":"55141f95-1bdc-4f48-acf2-8801738825bb","Type":"ContainerDied","Data":"749f05c5138aad75243b9ae481299194d0faee775f1f17b21f5653d702afae5a"}
Jul 29 22:03:03.110485 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.110444    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z" event={"ID":"55141f95-1bdc-4f48-acf2-8801738825bb","Type":"ContainerDied","Data":"1612f1b255e74902900f6f2cb5a89cb20851c95744bd8de6453b9e6bebc27218"}
Jul 29 22:03:03.110485 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.110466    2584 scope.go:117] "RemoveContainer" containerID="749f05c5138aad75243b9ae481299194d0faee775f1f17b21f5653d702afae5a"
Jul 29 22:03:03.111594 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.111564    2584 generic.go:356] "Generic (PLEG): container finished" podID="5cc377ca-cb0f-4bd1-a08b-7f700879a4f7" containerID="0113b3012486920365e0f4fc13e200a6ba3b7ab4f87bf866c7aa0ccf3ffc7d52" exitCode=0
Jul 29 22:03:03.111697 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.111611    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4" event={"ID":"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7","Type":"ContainerDied","Data":"0113b3012486920365e0f4fc13e200a6ba3b7ab4f87bf866c7aa0ccf3ffc7d52"}
Jul 29 22:03:03.111697 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.111617    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"
Jul 29 22:03:03.111697 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.111636    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4" event={"ID":"5cc377ca-cb0f-4bd1-a08b-7f700879a4f7","Type":"ContainerDied","Data":"2f61b152f8a091886dc3edf69db2c8ef3a9aeaf4afb2999050ca216f3f3bdfa6"}
Jul 29 22:03:03.118644 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.118625    2584 scope.go:117] "RemoveContainer" containerID="749f05c5138aad75243b9ae481299194d0faee775f1f17b21f5653d702afae5a"
Jul 29 22:03:03.119129 ip-10-0-129-194 kubenswrapper[2584]: E0729 22:03:03.119092    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749f05c5138aad75243b9ae481299194d0faee775f1f17b21f5653d702afae5a\": container with ID starting with 749f05c5138aad75243b9ae481299194d0faee775f1f17b21f5653d702afae5a not found: ID does not exist" containerID="749f05c5138aad75243b9ae481299194d0faee775f1f17b21f5653d702afae5a"
Jul 29 22:03:03.119260 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.119126    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749f05c5138aad75243b9ae481299194d0faee775f1f17b21f5653d702afae5a"} err="failed to get container status \"749f05c5138aad75243b9ae481299194d0faee775f1f17b21f5653d702afae5a\": rpc error: code = NotFound desc = could not find container \"749f05c5138aad75243b9ae481299194d0faee775f1f17b21f5653d702afae5a\": container with ID starting with 749f05c5138aad75243b9ae481299194d0faee775f1f17b21f5653d702afae5a not found: ID does not exist"
Jul 29 22:03:03.119260 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.119174    2584 scope.go:117] "RemoveContainer" containerID="0113b3012486920365e0f4fc13e200a6ba3b7ab4f87bf866c7aa0ccf3ffc7d52"
Jul 29 22:03:03.128579 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.128557    2584 scope.go:117] "RemoveContainer" containerID="0113b3012486920365e0f4fc13e200a6ba3b7ab4f87bf866c7aa0ccf3ffc7d52"
Jul 29 22:03:03.128880 ip-10-0-129-194 kubenswrapper[2584]: E0729 22:03:03.128854    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0113b3012486920365e0f4fc13e200a6ba3b7ab4f87bf866c7aa0ccf3ffc7d52\": container with ID starting with 0113b3012486920365e0f4fc13e200a6ba3b7ab4f87bf866c7aa0ccf3ffc7d52 not found: ID does not exist" containerID="0113b3012486920365e0f4fc13e200a6ba3b7ab4f87bf866c7aa0ccf3ffc7d52"
Jul 29 22:03:03.128931 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.128893    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0113b3012486920365e0f4fc13e200a6ba3b7ab4f87bf866c7aa0ccf3ffc7d52"} err="failed to get container status \"0113b3012486920365e0f4fc13e200a6ba3b7ab4f87bf866c7aa0ccf3ffc7d52\": rpc error: code = NotFound desc = could not find container \"0113b3012486920365e0f4fc13e200a6ba3b7ab4f87bf866c7aa0ccf3ffc7d52\": container with ID starting with 0113b3012486920365e0f4fc13e200a6ba3b7ab4f87bf866c7aa0ccf3ffc7d52 not found: ID does not exist"
Jul 29 22:03:03.132930 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.132907    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"]
Jul 29 22:03:03.136067 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.136045    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b69e52/rate-test-model-b69e52-kserve-d4d6b5687-49jq4"]
Jul 29 22:03:03.221024 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.220994    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55141f95-1bdc-4f48-acf2-8801738825bb-tls-certs\") pod \"55141f95-1bdc-4f48-acf2-8801738825bb\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") "
Jul 29 22:03:03.221241 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.221044    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-tmp-dir\") pod \"55141f95-1bdc-4f48-acf2-8801738825bb\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") "
Jul 29 22:03:03.221241 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.221079    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-model-cache\") pod \"55141f95-1bdc-4f48-acf2-8801738825bb\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") "
Jul 29 22:03:03.221241 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.221112    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-home\") pod \"55141f95-1bdc-4f48-acf2-8801738825bb\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") "
Jul 29 22:03:03.221241 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.221127    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-dshm\") pod \"55141f95-1bdc-4f48-acf2-8801738825bb\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") "
Jul 29 22:03:03.221241 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.221165    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn8pk\" (UniqueName: \"kubernetes.io/projected/55141f95-1bdc-4f48-acf2-8801738825bb-kube-api-access-fn8pk\") pod \"55141f95-1bdc-4f48-acf2-8801738825bb\" (UID: \"55141f95-1bdc-4f48-acf2-8801738825bb\") "
Jul 29 22:03:03.221523 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.221391    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55141f95-1bdc-4f48-acf2-8801738825bb" (UID: "55141f95-1bdc-4f48-acf2-8801738825bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:03.221523 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.221419    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-home" (OuterVolumeSpecName: "home") pod "55141f95-1bdc-4f48-acf2-8801738825bb" (UID: "55141f95-1bdc-4f48-acf2-8801738825bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:03.221523 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.221476    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "55141f95-1bdc-4f48-acf2-8801738825bb" (UID: "55141f95-1bdc-4f48-acf2-8801738825bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:03.223200 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.223177    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55141f95-1bdc-4f48-acf2-8801738825bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55141f95-1bdc-4f48-acf2-8801738825bb" (UID: "55141f95-1bdc-4f48-acf2-8801738825bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:03:03.223312 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.223293    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-dshm" (OuterVolumeSpecName: "dshm") pod "55141f95-1bdc-4f48-acf2-8801738825bb" (UID: "55141f95-1bdc-4f48-acf2-8801738825bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:03.223375 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.223338    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55141f95-1bdc-4f48-acf2-8801738825bb-kube-api-access-fn8pk" (OuterVolumeSpecName: "kube-api-access-fn8pk") pod "55141f95-1bdc-4f48-acf2-8801738825bb" (UID: "55141f95-1bdc-4f48-acf2-8801738825bb"). InnerVolumeSpecName "kube-api-access-fn8pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:03.321993 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.321949    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-tmp-dir\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:03.321993 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.321984    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-model-cache\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:03.321993 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.321993    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-home\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:03.321993 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.322002    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55141f95-1bdc-4f48-acf2-8801738825bb-dshm\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:03.322323 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.322011    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn8pk\" (UniqueName: \"kubernetes.io/projected/55141f95-1bdc-4f48-acf2-8801738825bb-kube-api-access-fn8pk\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:03.322323 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.322021    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55141f95-1bdc-4f48-acf2-8801738825bb-tls-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:03.434893 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.434858    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"]
Jul 29 22:03:03.438617 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:03.438590    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ac47f7/rate-test-model-ac47f7-kserve-5b8544c5d7-rl77z"]
Jul 29 22:03:04.865710 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:04.865677    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55141f95-1bdc-4f48-acf2-8801738825bb" path="/var/lib/kubelet/pods/55141f95-1bdc-4f48-acf2-8801738825bb/volumes"
Jul 29 22:03:04.866494 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:04.866468    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cc377ca-cb0f-4bd1-a08b-7f700879a4f7" path="/var/lib/kubelet/pods/5cc377ca-cb0f-4bd1-a08b-7f700879a4f7/volumes"
Jul 29 22:03:05.101176 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:05.101125    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"]
Jul 29 22:03:05.101380 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:05.101355    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt" podUID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" containerName="istio-proxy" containerID="cri-o://d81aa21c28770cdd837d1a2eb2c0e240b8f0a763da17e4c2017e19424be7bbea" gracePeriod=30
Jul 29 22:03:10.353959 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.353936    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 22:03:10.379351 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.379279    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stvc5\" (UniqueName: \"kubernetes.io/projected/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-kube-api-access-stvc5\") pod \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") "
Jul 29 22:03:10.379351 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.379332    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-podinfo\") pod \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") "
Jul 29 22:03:10.379563 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.379361    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-workload-certs\") pod \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") "
Jul 29 22:03:10.379563 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.379384    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-workload-socket\") pod \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") "
Jul 29 22:03:10.379563 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.379412    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-data\") pod \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") "
Jul 29 22:03:10.379563 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.379476    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istiod-ca-cert\") pod \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") "
Jul 29 22:03:10.379563 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.379502    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-credential-socket\") pod \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") "
Jul 29 22:03:10.379563 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.379539    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-token\") pod \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") "
Jul 29 22:03:10.380277 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.379566    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-envoy\") pod \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\" (UID: \"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed\") "
Jul 29 22:03:10.380277 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.379885    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" (UID: "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:10.380277 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.379921    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" (UID: "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:03:10.380277 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.379926    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" (UID: "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:10.380277 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.380029    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" (UID: "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:10.380553 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.380398    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-data" (OuterVolumeSpecName: "istio-data") pod "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" (UID: "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:10.382020 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.381891    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" (UID: "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 22:03:10.382020 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.381891    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" (UID: "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:10.382020 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.381995    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-kube-api-access-stvc5" (OuterVolumeSpecName: "kube-api-access-stvc5") pod "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" (UID: "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed"). InnerVolumeSpecName "kube-api-access-stvc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:10.382320 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.382165    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" (UID: "fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:10.480657 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.480619    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stvc5\" (UniqueName: \"kubernetes.io/projected/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-kube-api-access-stvc5\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:10.480657 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.480650    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:10.480657 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.480660    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:10.480657 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.480669    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:10.480932 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.480678    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:10.480932 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.480688    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:10.480932 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.480697    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:10.480932 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.480704    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:10.480932 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:10.480712    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:11.150607 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:11.150509    2584 generic.go:356] "Generic (PLEG): container finished" podID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" containerID="d81aa21c28770cdd837d1a2eb2c0e240b8f0a763da17e4c2017e19424be7bbea" exitCode=0
Jul 29 22:03:11.150607 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:11.150556    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"
Jul 29 22:03:11.150820 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:11.150598    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt" event={"ID":"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed","Type":"ContainerDied","Data":"d81aa21c28770cdd837d1a2eb2c0e240b8f0a763da17e4c2017e19424be7bbea"}
Jul 29 22:03:11.150820 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:11.150658    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt" event={"ID":"fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed","Type":"ContainerDied","Data":"30a13618d70bb2f756602dbe95ff2b4092bfbd5811caac39df124de8d635c870"}
Jul 29 22:03:11.150820 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:11.150682    2584 scope.go:117] "RemoveContainer" containerID="d81aa21c28770cdd837d1a2eb2c0e240b8f0a763da17e4c2017e19424be7bbea"
Jul 29 22:03:11.159042 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:11.159023    2584 scope.go:117] "RemoveContainer" containerID="e4802ff0b4bcadf991d3b29b76f0e0312eadc4412f90bdf00cd5614f7245c41c"
Jul 29 22:03:11.166414 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:11.166390    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"]
Jul 29 22:03:11.167078 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:11.167061    2584 scope.go:117] "RemoveContainer" containerID="d81aa21c28770cdd837d1a2eb2c0e240b8f0a763da17e4c2017e19424be7bbea"
Jul 29 22:03:11.167347 ip-10-0-129-194 kubenswrapper[2584]: E0729 22:03:11.167329    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81aa21c28770cdd837d1a2eb2c0e240b8f0a763da17e4c2017e19424be7bbea\": container with ID starting with d81aa21c28770cdd837d1a2eb2c0e240b8f0a763da17e4c2017e19424be7bbea not found: ID does not exist" containerID="d81aa21c28770cdd837d1a2eb2c0e240b8f0a763da17e4c2017e19424be7bbea"
Jul 29 22:03:11.167413 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:11.167353    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81aa21c28770cdd837d1a2eb2c0e240b8f0a763da17e4c2017e19424be7bbea"} err="failed to get container status \"d81aa21c28770cdd837d1a2eb2c0e240b8f0a763da17e4c2017e19424be7bbea\": rpc error: code = NotFound desc = could not find container \"d81aa21c28770cdd837d1a2eb2c0e240b8f0a763da17e4c2017e19424be7bbea\": container with ID starting with d81aa21c28770cdd837d1a2eb2c0e240b8f0a763da17e4c2017e19424be7bbea not found: ID does not exist"
Jul 29 22:03:11.167413 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:11.167370    2584 scope.go:117] "RemoveContainer" containerID="e4802ff0b4bcadf991d3b29b76f0e0312eadc4412f90bdf00cd5614f7245c41c"
Jul 29 22:03:11.167563 ip-10-0-129-194 kubenswrapper[2584]: E0729 22:03:11.167544    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4802ff0b4bcadf991d3b29b76f0e0312eadc4412f90bdf00cd5614f7245c41c\": container with ID starting with e4802ff0b4bcadf991d3b29b76f0e0312eadc4412f90bdf00cd5614f7245c41c not found: ID does not exist" containerID="e4802ff0b4bcadf991d3b29b76f0e0312eadc4412f90bdf00cd5614f7245c41c"
Jul 29 22:03:11.167612 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:11.167574    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4802ff0b4bcadf991d3b29b76f0e0312eadc4412f90bdf00cd5614f7245c41c"} err="failed to get container status \"e4802ff0b4bcadf991d3b29b76f0e0312eadc4412f90bdf00cd5614f7245c41c\": rpc error: code = NotFound desc = could not find container \"e4802ff0b4bcadf991d3b29b76f0e0312eadc4412f90bdf00cd5614f7245c41c\": container with ID starting with e4802ff0b4bcadf991d3b29b76f0e0312eadc4412f90bdf00cd5614f7245c41c not found: ID does not exist"
Jul 29 22:03:11.170893 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:11.170871    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-b69e52-openshift-default-7f4949bbc6-lz4qt"]
Jul 29 22:03:12.865680 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:12.865649    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" path="/var/lib/kubelet/pods/fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed/volumes"
Jul 29 22:03:22.533703 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:22.533668    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"]
Jul 29 22:03:22.534314 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:22.533933    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7" podUID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerName="istio-proxy" containerID="cri-o://842af4df9891350ac834d50e3d325c12889aa1aeb058ad5122f1ffafa097ba9a" gracePeriod=30
Jul 29 22:03:23.247730 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:23.247694    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54cd85c6d4-lpd8s_8f405d7a-0815-4971-9df0-1d981d95da5f/authorino/0.log"
Jul 29 22:03:27.791334 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.791309    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 22:03:27.932104 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932070    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-credential-socket\") pod \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") "
Jul 29 22:03:27.932104 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932108    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtsp4\" (UniqueName: \"kubernetes.io/projected/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-kube-api-access-vtsp4\") pod \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") "
Jul 29 22:03:27.932348 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932125    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-podinfo\") pod \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") "
Jul 29 22:03:27.932348 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932182    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-token\") pod \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") "
Jul 29 22:03:27.932348 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932305    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istiod-ca-cert\") pod \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") "
Jul 29 22:03:27.932485 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932351    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-data\") pod \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") "
Jul 29 22:03:27.932485 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932398    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-envoy\") pod \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") "
Jul 29 22:03:27.932485 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932430    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" (UID: "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:27.932485 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932462    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-workload-socket\") pod \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") "
Jul 29 22:03:27.932700 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932488    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-workload-certs\") pod \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\" (UID: \"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f\") "
Jul 29 22:03:27.932765 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932712    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" (UID: "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:27.932765 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932739    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-credential-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:27.932765 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932740    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" (UID: "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:03:27.932959 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.932934    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" (UID: "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:27.933393 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.933370    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-data" (OuterVolumeSpecName: "istio-data") pod "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" (UID: "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:27.934410 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.934388    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-token" (OuterVolumeSpecName: "istio-token") pod "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" (UID: "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:27.934512 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.934473    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" (UID: "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:27.934744 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.934727    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" (UID: "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 22:03:27.934810 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:27.934726    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-kube-api-access-vtsp4" (OuterVolumeSpecName: "kube-api-access-vtsp4") pod "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" (UID: "701b1061-a6b2-45d2-bd2e-e05dd32a7f5f"). InnerVolumeSpecName "kube-api-access-vtsp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:28.033898 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.033861    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtsp4\" (UniqueName: \"kubernetes.io/projected/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-kube-api-access-vtsp4\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:28.033898 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.033894    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-podinfo\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:28.033898 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.033905    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-token\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:28.034136 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.033915    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istiod-ca-cert\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:28.034136 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.033923    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-data\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:28.034136 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.033931    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-istio-envoy\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:28.034136 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.033939    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-workload-socket\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:28.034136 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.033948    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f-workload-certs\") on node \"ip-10-0-129-194.ec2.internal\" DevicePath \"\""
Jul 29 22:03:28.225745 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.225664    2584 generic.go:356] "Generic (PLEG): container finished" podID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerID="842af4df9891350ac834d50e3d325c12889aa1aeb058ad5122f1ffafa097ba9a" exitCode=0
Jul 29 22:03:28.225745 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.225707    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"
Jul 29 22:03:28.225927 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.225744    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7" event={"ID":"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f","Type":"ContainerDied","Data":"842af4df9891350ac834d50e3d325c12889aa1aeb058ad5122f1ffafa097ba9a"}
Jul 29 22:03:28.225927 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.225803    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7" event={"ID":"701b1061-a6b2-45d2-bd2e-e05dd32a7f5f","Type":"ContainerDied","Data":"bd7c2f7c695a456ce5f90b9914ded7ce255015264388a9953cad336869da04d0"}
Jul 29 22:03:28.225927 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.225826    2584 scope.go:117] "RemoveContainer" containerID="842af4df9891350ac834d50e3d325c12889aa1aeb058ad5122f1ffafa097ba9a"
Jul 29 22:03:28.234411 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.234389    2584 scope.go:117] "RemoveContainer" containerID="d7a20ac010025973bd059176e625a215f9733b615e1b70b7b5b72ecd4c2b1dd2"
Jul 29 22:03:28.241985 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.241969    2584 scope.go:117] "RemoveContainer" containerID="842af4df9891350ac834d50e3d325c12889aa1aeb058ad5122f1ffafa097ba9a"
Jul 29 22:03:28.242283 ip-10-0-129-194 kubenswrapper[2584]: E0729 22:03:28.242260    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842af4df9891350ac834d50e3d325c12889aa1aeb058ad5122f1ffafa097ba9a\": container with ID starting with 842af4df9891350ac834d50e3d325c12889aa1aeb058ad5122f1ffafa097ba9a not found: ID does not exist" containerID="842af4df9891350ac834d50e3d325c12889aa1aeb058ad5122f1ffafa097ba9a"
Jul 29 22:03:28.242340 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.242293    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842af4df9891350ac834d50e3d325c12889aa1aeb058ad5122f1ffafa097ba9a"} err="failed to get container status \"842af4df9891350ac834d50e3d325c12889aa1aeb058ad5122f1ffafa097ba9a\": rpc error: code = NotFound desc = could not find container \"842af4df9891350ac834d50e3d325c12889aa1aeb058ad5122f1ffafa097ba9a\": container with ID starting with 842af4df9891350ac834d50e3d325c12889aa1aeb058ad5122f1ffafa097ba9a not found: ID does not exist"
Jul 29 22:03:28.242340 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.242311    2584 scope.go:117] "RemoveContainer" containerID="d7a20ac010025973bd059176e625a215f9733b615e1b70b7b5b72ecd4c2b1dd2"
Jul 29 22:03:28.242540 ip-10-0-129-194 kubenswrapper[2584]: E0729 22:03:28.242522    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7a20ac010025973bd059176e625a215f9733b615e1b70b7b5b72ecd4c2b1dd2\": container with ID starting with d7a20ac010025973bd059176e625a215f9733b615e1b70b7b5b72ecd4c2b1dd2 not found: ID does not exist" containerID="d7a20ac010025973bd059176e625a215f9733b615e1b70b7b5b72ecd4c2b1dd2"
Jul 29 22:03:28.242605 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.242550    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a20ac010025973bd059176e625a215f9733b615e1b70b7b5b72ecd4c2b1dd2"} err="failed to get container status \"d7a20ac010025973bd059176e625a215f9733b615e1b70b7b5b72ecd4c2b1dd2\": rpc error: code = NotFound desc = could not find container \"d7a20ac010025973bd059176e625a215f9733b615e1b70b7b5b72ecd4c2b1dd2\": container with ID starting with d7a20ac010025973bd059176e625a215f9733b615e1b70b7b5b72ecd4c2b1dd2 not found: ID does not exist"
Jul 29 22:03:28.247102 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.247079    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"]
Jul 29 22:03:28.251095 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.251073    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-ac47f7-openshift-default-555d77f668-2j9s7"]
Jul 29 22:03:28.517057 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.516973    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8x7hm_939915a3-8bc5-4be2-a221-f7ca7ff44d94/manager/0.log"
Jul 29 22:03:28.644879 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.644842    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lgk7z_779aa379-51b0-430b-97f2-7f53d397fcec/manager/0.log"
Jul 29 22:03:28.768906 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.768824    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c85847677-c7fqc_6f3f9a7e-2441-476a-8f8d-1b72f46a2384/manager/0.log"
Jul 29 22:03:28.870553 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:28.870519    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" path="/var/lib/kubelet/pods/701b1061-a6b2-45d2-bd2e-e05dd32a7f5f/volumes"
Jul 29 22:03:29.378571 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:29.378543    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dbc94cc7-jgz46_f820a960-882d-488f-8de5-3aaa08ace55f/copy-manifests/0.log"
Jul 29 22:03:29.385531 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:29.385493    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dbc94cc7-jgz46_f820a960-882d-488f-8de5-3aaa08ace55f/manager/0.log"
Jul 29 22:03:29.848348 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:29.848315    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756010-kbj6g_24497491-89d5-4128-8ac7-c50e65201d08/cleanup/0.log"
Jul 29 22:03:29.964819 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:29.964793    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756025-9rr4b_7f453691-7fea-4373-86b0-316dd0c09139/cleanup/0.log"
Jul 29 22:03:31.489245 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:31.489211    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54cd85c6d4-lpd8s_8f405d7a-0815-4971-9df0-1d981d95da5f/authorino/0.log"
Jul 29 22:03:31.607389 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:31.607350    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5wrts_b406c2b2-6add-4e70-bdaa-c358471c2144/manager/0.log"
Jul 29 22:03:32.320497 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:32.320444    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x5mwx_6388a4d7-6919-407d-9dc4-c41a357e86af/manager/0.log"
Jul 29 22:03:32.660080 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:32.660043    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc_e8e266ed-b7ab-4356-83c4-b942b5350187/istio-proxy/0.log"
Jul 29 22:03:34.031711 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:34.031681    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-gkmlq_b96d1ae2-829d-4164-a167-7c6ce09b109e/main/0.log"
Jul 29 22:03:39.535783 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.535748    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-crrnx/must-gather-tvbh7"]
Jul 29 22:03:39.536201 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536065    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" containerName="istio-proxy"
Jul 29 22:03:39.536201 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536075    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" containerName="istio-proxy"
Jul 29 22:03:39.536201 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536088    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerName="istio-proxy"
Jul 29 22:03:39.536201 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536093    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerName="istio-proxy"
Jul 29 22:03:39.536201 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536101    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55141f95-1bdc-4f48-acf2-8801738825bb" containerName="main"
Jul 29 22:03:39.536201 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536107    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="55141f95-1bdc-4f48-acf2-8801738825bb" containerName="main"
Jul 29 22:03:39.536201 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536117    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc377ca-cb0f-4bd1-a08b-7f700879a4f7" containerName="main"
Jul 29 22:03:39.536201 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536122    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc377ca-cb0f-4bd1-a08b-7f700879a4f7" containerName="main"
Jul 29 22:03:39.536201 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536134    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerName="istio-proxy"
Jul 29 22:03:39.536201 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536139    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerName="istio-proxy"
Jul 29 22:03:39.536201 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536172    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" containerName="istio-proxy"
Jul 29 22:03:39.536201 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536179    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" containerName="istio-proxy"
Jul 29 22:03:39.536627 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536236    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cc377ca-cb0f-4bd1-a08b-7f700879a4f7" containerName="main"
Jul 29 22:03:39.536627 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536244    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="55141f95-1bdc-4f48-acf2-8801738825bb" containerName="main"
Jul 29 22:03:39.536627 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536253    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerName="istio-proxy"
Jul 29 22:03:39.536627 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536260    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" containerName="istio-proxy"
Jul 29 22:03:39.536627 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536265    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdfeb8cf-dd51-4bf9-aede-b37f29f4b8ed" containerName="istio-proxy"
Jul 29 22:03:39.536627 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.536271    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="701b1061-a6b2-45d2-bd2e-e05dd32a7f5f" containerName="istio-proxy"
Jul 29 22:03:39.540585 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.540567    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crrnx/must-gather-tvbh7"
Jul 29 22:03:39.543271 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.543250    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-crrnx\"/\"default-dockercfg-rf2bz\""
Jul 29 22:03:39.545187 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.545168    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-crrnx\"/\"kube-root-ca.crt\""
Jul 29 22:03:39.545282 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.545214    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-crrnx\"/\"openshift-service-ca.crt\""
Jul 29 22:03:39.552168 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.552118    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crrnx/must-gather-tvbh7"]
Jul 29 22:03:39.633723 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.633685    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2srws\" (UniqueName: \"kubernetes.io/projected/59e9c73d-3317-4c3a-9e4f-1bc2789833d1-kube-api-access-2srws\") pod \"must-gather-tvbh7\" (UID: \"59e9c73d-3317-4c3a-9e4f-1bc2789833d1\") " pod="openshift-must-gather-crrnx/must-gather-tvbh7"
Jul 29 22:03:39.633901 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.633747    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59e9c73d-3317-4c3a-9e4f-1bc2789833d1-must-gather-output\") pod \"must-gather-tvbh7\" (UID: \"59e9c73d-3317-4c3a-9e4f-1bc2789833d1\") " pod="openshift-must-gather-crrnx/must-gather-tvbh7"
Jul 29 22:03:39.734914 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.734875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2srws\" (UniqueName: \"kubernetes.io/projected/59e9c73d-3317-4c3a-9e4f-1bc2789833d1-kube-api-access-2srws\") pod \"must-gather-tvbh7\" (UID: \"59e9c73d-3317-4c3a-9e4f-1bc2789833d1\") " pod="openshift-must-gather-crrnx/must-gather-tvbh7"
Jul 29 22:03:39.735066 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.734935    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59e9c73d-3317-4c3a-9e4f-1bc2789833d1-must-gather-output\") pod \"must-gather-tvbh7\" (UID: \"59e9c73d-3317-4c3a-9e4f-1bc2789833d1\") " pod="openshift-must-gather-crrnx/must-gather-tvbh7"
Jul 29 22:03:39.735289 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.735271    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59e9c73d-3317-4c3a-9e4f-1bc2789833d1-must-gather-output\") pod \"must-gather-tvbh7\" (UID: \"59e9c73d-3317-4c3a-9e4f-1bc2789833d1\") " pod="openshift-must-gather-crrnx/must-gather-tvbh7"
Jul 29 22:03:39.747909 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.747879    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2srws\" (UniqueName: \"kubernetes.io/projected/59e9c73d-3317-4c3a-9e4f-1bc2789833d1-kube-api-access-2srws\") pod \"must-gather-tvbh7\" (UID: \"59e9c73d-3317-4c3a-9e4f-1bc2789833d1\") " pod="openshift-must-gather-crrnx/must-gather-tvbh7"
Jul 29 22:03:39.849777 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.849746    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crrnx/must-gather-tvbh7"
Jul 29 22:03:39.983515 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.983478    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crrnx/must-gather-tvbh7"]
Jul 29 22:03:39.986712 ip-10-0-129-194 kubenswrapper[2584]: W0729 22:03:39.986684    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59e9c73d_3317_4c3a_9e4f_1bc2789833d1.slice/crio-1a6db177afde0708083acf6e50a235bd0e29085fcaa6f670d0f0b11c11eaf95b WatchSource:0}: Error finding container 1a6db177afde0708083acf6e50a235bd0e29085fcaa6f670d0f0b11c11eaf95b: Status 404 returned error can't find the container with id 1a6db177afde0708083acf6e50a235bd0e29085fcaa6f670d0f0b11c11eaf95b
Jul 29 22:03:39.989093 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:39.989076    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 22:03:40.276791 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:40.276697    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crrnx/must-gather-tvbh7" event={"ID":"59e9c73d-3317-4c3a-9e4f-1bc2789833d1","Type":"ContainerStarted","Data":"1a6db177afde0708083acf6e50a235bd0e29085fcaa6f670d0f0b11c11eaf95b"}
Jul 29 22:03:42.287772 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:42.287729    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crrnx/must-gather-tvbh7" event={"ID":"59e9c73d-3317-4c3a-9e4f-1bc2789833d1","Type":"ContainerStarted","Data":"c4b1b497e5bc2493fded228765b4f336bef51efee116fc5c3316e97e983afda4"}
Jul 29 22:03:42.288213 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:42.287782    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crrnx/must-gather-tvbh7" event={"ID":"59e9c73d-3317-4c3a-9e4f-1bc2789833d1","Type":"ContainerStarted","Data":"044946ac3bac32364b9218a2a1b629ad683060b1f853a874379bfb7c4992035a"}
Jul 29 22:03:42.303509 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:42.303460    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-crrnx/must-gather-tvbh7" podStartSLOduration=1.805556328 podStartE2EDuration="3.303443944s" totalImagesPullingTime="1.497887616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:03:42.300971487 +0000 UTC m=+4637.897263181" watchObservedRunningTime="2026-07-29 22:03:42.303443944 +0000 UTC m=+4637.899735627"
Jul 29 22:03:43.117222 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:43.117188    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rzcl7_eeebe086-c70d-4f85-afbb-5e159fe76372/global-pull-secret-syncer/0.log"
Jul 29 22:03:43.211604 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:43.211569    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m4fbv_67814655-dde0-40dc-b521-b4242f0cec15/konnectivity-agent/0.log"
Jul 29 22:03:43.275704 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:43.275673    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-194.ec2.internal_5d5318fce92e6dc30124e077fbe98465/haproxy/0.log"
Jul 29 22:03:48.481698 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:48.481639    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54cd85c6d4-lpd8s_8f405d7a-0815-4971-9df0-1d981d95da5f/authorino/0.log"
Jul 29 22:03:48.521075 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:48.520893    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5wrts_b406c2b2-6add-4e70-bdaa-c358471c2144/manager/0.log"
Jul 29 22:03:48.877163 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:48.875965    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x5mwx_6388a4d7-6919-407d-9dc4-c41a357e86af/manager/0.log"
Jul 29 22:03:51.009101 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:51.009064    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hnqdd_f1f85408-f650-4bb2-84c5-6943cefc4c5f/node-exporter/0.log"
Jul 29 22:03:51.032248 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:51.032183    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hnqdd_f1f85408-f650-4bb2-84c5-6943cefc4c5f/kube-rbac-proxy/0.log"
Jul 29 22:03:51.060408 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:51.060376    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hnqdd_f1f85408-f650-4bb2-84c5-6943cefc4c5f/init-textfile/0.log"
Jul 29 22:03:51.188907 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:51.188876    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-26pn4_8c4999cf-05ac-4056-82ff-9917e356f0bb/kube-rbac-proxy-main/0.log"
Jul 29 22:03:51.214766 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:51.214737    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-26pn4_8c4999cf-05ac-4056-82ff-9917e356f0bb/kube-rbac-proxy-self/0.log"
Jul 29 22:03:51.240490 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:51.240458    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-26pn4_8c4999cf-05ac-4056-82ff-9917e356f0bb/openshift-state-metrics/0.log"
Jul 29 22:03:51.569095 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:51.569068    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f56848b96-sqctc_5e7648fb-e8b3-46ff-8bed-1e071c2acd38/telemeter-client/0.log"
Jul 29 22:03:51.594336 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:51.594304    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f56848b96-sqctc_5e7648fb-e8b3-46ff-8bed-1e071c2acd38/reload/0.log"
Jul 29 22:03:51.617869 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:51.617835    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f56848b96-sqctc_5e7648fb-e8b3-46ff-8bed-1e071c2acd38/kube-rbac-proxy/0.log"
Jul 29 22:03:53.205522 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.205478    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"]
Jul 29 22:03:53.211887 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.211856    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.221065 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.221037    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"]
Jul 29 22:03:53.277372 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.277341    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-lib-modules\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.277566 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.277459    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-podres\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.277566 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.277500    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-sys\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.277566 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.277545    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-proc\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.277741 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.277622    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4nd6\" (UniqueName: \"kubernetes.io/projected/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-kube-api-access-l4nd6\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.378502 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.378458    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-lib-modules\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.378502 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.378507    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-podres\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.378763 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.378529    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-sys\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.378763 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.378638    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-lib-modules\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.378763 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.378671    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-podres\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.378763 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.378689    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-proc\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.378934 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.378776    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4nd6\" (UniqueName: \"kubernetes.io/projected/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-kube-api-access-l4nd6\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.378934 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.378794    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-sys\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.378934 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.378867    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-proc\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.387297 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.387238    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4nd6\" (UniqueName: \"kubernetes.io/projected/7cb2e35c-4d7e-4bb7-a712-a4457a7db746-kube-api-access-l4nd6\") pod \"perf-node-gather-daemonset-rvxdn\" (UID: \"7cb2e35c-4d7e-4bb7-a712-a4457a7db746\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.530175 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.530060    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:53.703757 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:53.701649    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"]
Jul 29 22:03:54.367984 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:54.367950    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn" event={"ID":"7cb2e35c-4d7e-4bb7-a712-a4457a7db746","Type":"ContainerStarted","Data":"eb1c62f61e853bfbb2cb2b5ab971a5914e0ec7479f2073785bf91970a15df8b5"}
Jul 29 22:03:54.368603 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:54.368584    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:03:54.368944 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:54.368925    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn" event={"ID":"7cb2e35c-4d7e-4bb7-a712-a4457a7db746","Type":"ContainerStarted","Data":"e27bec793fbae626f124e5679016108bac742620d43b5c655b4a22a58d17665a"}
Jul 29 22:03:54.385230 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:54.385173    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn" podStartSLOduration=1.3851385 podStartE2EDuration="1.3851385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:03:54.382662416 +0000 UTC m=+4649.978954125" watchObservedRunningTime="2026-07-29 22:03:54.3851385 +0000 UTC m=+4649.981430185"
Jul 29 22:03:55.774534 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:55.774502    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d272j_53634e55-d5a1-44f7-baff-f9183c1fc7bd/dns/0.log"
Jul 29 22:03:55.795035 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:55.794997    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d272j_53634e55-d5a1-44f7-baff-f9183c1fc7bd/kube-rbac-proxy/0.log"
Jul 29 22:03:55.912108 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:55.912080    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2z8d_4b75e3fd-cc2d-4da4-9026-c8b6302dae66/dns-node-resolver/0.log"
Jul 29 22:03:56.520799 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:56.520766    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-562n4_042d3857-ba4b-4d0d-b9da-3ad4d6003cd3/node-ca/0.log"
Jul 29 22:03:57.503882 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:57.503846    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc_e8e266ed-b7ab-4356-83c4-b942b5350187/istio-proxy/0.log"
Jul 29 22:03:58.661423 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:58.661392    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qnhjk_5c357a9f-b01a-41c9-b36d-cdd506c9b60d/serve-healthcheck-canary/0.log"
Jul 29 22:03:59.219984 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:59.219955    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xldl_1ecfbe60-3617-43c2-bacf-f347a1e88b8c/kube-rbac-proxy/0.log"
Jul 29 22:03:59.242365 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:59.242340    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xldl_1ecfbe60-3617-43c2-bacf-f347a1e88b8c/exporter/0.log"
Jul 29 22:03:59.266674 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:03:59.266634    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xldl_1ecfbe60-3617-43c2-bacf-f347a1e88b8c/extractor/0.log"
Jul 29 22:04:01.390026 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:01.389999    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-rvxdn"
Jul 29 22:04:01.436591 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:01.436556    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8x7hm_939915a3-8bc5-4be2-a221-f7ca7ff44d94/manager/0.log"
Jul 29 22:04:01.862944 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:01.862908    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lgk7z_779aa379-51b0-430b-97f2-7f53d397fcec/manager/0.log"
Jul 29 22:04:02.192360 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:02.192282    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c85847677-c7fqc_6f3f9a7e-2441-476a-8f8d-1b72f46a2384/manager/0.log"
Jul 29 22:04:02.490293 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:02.490210    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dbc94cc7-jgz46_f820a960-882d-488f-8de5-3aaa08ace55f/manager/0.log"
Jul 29 22:04:02.511527 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:02.511500    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dbc94cc7-jgz46_f820a960-882d-488f-8de5-3aaa08ace55f/copy-manifests/0.log"
Jul 29 22:04:03.868819 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:03.868777    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6896cc89b8-kfd7t_e26c4b45-4911-482f-9815-e4c1140b68e4/manager/0.log"
Jul 29 22:04:03.940353 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:03.940320    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc_e8e266ed-b7ab-4356-83c4-b942b5350187/istio-proxy/0.log"
Jul 29 22:04:05.599505 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:05.599476    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-967mc_e8e266ed-b7ab-4356-83c4-b942b5350187/istio-proxy/0.log"
Jul 29 22:04:10.294947 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:10.294906    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7cxkg_df049e92-c4e8-449f-9dd1-8f14e7665d2b/kube-multus/0.log"
Jul 29 22:04:10.317805 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:10.317782    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmrct_c4bb2f8a-22d1-4fd5-9722-81d47ada14f1/kube-multus-additional-cni-plugins/0.log"
Jul 29 22:04:10.338599 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:10.338580    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmrct_c4bb2f8a-22d1-4fd5-9722-81d47ada14f1/egress-router-binary-copy/0.log"
Jul 29 22:04:10.361110 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:10.361090    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmrct_c4bb2f8a-22d1-4fd5-9722-81d47ada14f1/cni-plugins/0.log"
Jul 29 22:04:10.381271 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:10.381254    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmrct_c4bb2f8a-22d1-4fd5-9722-81d47ada14f1/bond-cni-plugin/0.log"
Jul 29 22:04:10.405749 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:10.405722    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmrct_c4bb2f8a-22d1-4fd5-9722-81d47ada14f1/routeoverride-cni/0.log"
Jul 29 22:04:10.427359 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:10.427333    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmrct_c4bb2f8a-22d1-4fd5-9722-81d47ada14f1/whereabouts-cni-bincopy/0.log"
Jul 29 22:04:10.448418 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:10.448391    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cmrct_c4bb2f8a-22d1-4fd5-9722-81d47ada14f1/whereabouts-cni/0.log"
Jul 29 22:04:10.918794 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:10.918758    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hdvwv_f4edb7a0-def8-488e-a6ea-eca10d6f1452/network-metrics-daemon/0.log"
Jul 29 22:04:10.936585 ip-10-0-129-194 kubenswrapper[2584]: I0729 22:04:10.936561    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hdvwv_f4edb7a0-def8-488e-a6ea-eca10d6f1452/kube-rbac-proxy/0.log"